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ABSTRACT 
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Australian requirements and conditions was investigated. Focus was 
on: the program's mission, capability, and setting; its management; 
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its development activities; its potential for relationships with 
other institutions, agencies and organizations; its schedule; its 
facilities; its funding; and its product dissemination. The 
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FOREWORD 



A period of two and a half years has elapsed between the conception of the gerniinal idea 
for this project and the publication of its consequent report. That idea was bom during 
the course of correspondence between the author and P. Kenneth Komoski, the 
Executive Director of the Educational Products Information Exchange (EPIE) Institute 
at New York, N.Y. during August 1986. The development of this project saw this idea 
transferred from that original source to its consideration for adoption at the national level 
of Australian education, and the collection of data on the problem area in Tasmanian 
education, and ^x an international level in the United States of America, in Canada, in 
the United Kinglom, in Iceland and in the Peoples Republic of China. 
More importandy, a veritable revolution has occurred in Australian education during this 
period. This extensive change commenced with the publication of The Challenge for 
Higher Education in Australia by John S. Dawkins, the Minister for Employment, 
Education and Training in September 1987. This statement was soon followed by die 
declaration of proposals to overhaul higher education presented in the Green Paper, 
entitled Higher Education: A Policy Discussion Paper. The more contentious proposals, 
particularly on die funding of higher education by die student population, formed 
controversial issues for debate in educational circles across the nation during 1987 and 
1988. 

At the same time, measures were also taken to restructure national educational 
organisations to effect changes in primary and secondary education so that these levels 
would meet more closely die needs of higher education. The Commonwealth Schools 
Commission was amalgamated within the Department of Employment, Education and 
Training during November 1987, and a National Board of Employment, Education and 
Training was formed. These organisational changes were also made in conjunction with 
the development of a national curriculum and assessment framework for Australian 
schools, first announced on 23 May 1988 by die Minister for Employment, Education and 
Training in the paper. Strengthening Australia' s Schools: A Consideration of the Focus and 
Content of Schooling. On 27 July 1988, the Minister for Employment, Education and 
Training, meeting togetiier witii die state and territorial ministers of education under the 
auspices of the Australian Education Council, agreed to work towards implementing a 
common curriculum for Australian schools in the core content areas of language arts, 
mathematics and sciences. Fiulhermore, the members of the Australian Education 
Council agreed that new structures and collaborative arrangements should be developed 
between national and state educational organisations to facilitate the implementation of 
the national curriculum and assessment framework. 

In order to examine these issues, the Australian Education Council formed two working 
parties: the first to develop a timeline for implementing the national curriculum 
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framework; and the second to determine the capabilities and the collaborative 
arrangements between the Australian Council for Educational Research (ACER), the 
Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS). A report from the latter working party is currently under 
consideration by the Australian Education Council, and a decision is expected to be 
announced during April 1989. 

These developments should be taken into account when reading this report. As this report 
was written between September 1987 and December 1988, the relationships between 
Australiiui educational organisations described in the report are those existing during this 
period. As a result of the work of the Australian Education Council during 1988 and 1989, 
the structures of these Australian educational organisations and their collaborative 
relationships may alter drastically after April 1989. 

In spite of these qualifications, the author presents this report to the Curriculum 
Development Centre and the Australian Schools Catalogue Information Service and to 
the representatives of foreign educational organisations as being consonant with the 
current changes occurring in the national planning of both the curriculum for Australian 
schools and the collaborative relationships between organisations in Australian 
education. 

Michael G. Watt 

Education Department of Tasmania 

Hobart, Tasmania 

Australia 



12 December 1988 



PREFACE 



Only with the assistance of a number of people was it possible to establish valid aims and 
directions to complete this report. The supervisor, Phillip W. Hughes, Professor of 
Education, University of Tasmania, and the experienced teachers participating in the 
Centre for Continuing Education of Teacher's course. Curriculum Workshop, are to be 
thanked for their contributions. 

The author wishes to acknowledge the assistance given by several colleagues of the 
Education Department of Tasmania in facilitating this project. Foremost, Glenn C. 
PuUen, Supervisor of Libraries, is to be thanked for providing the author with the reason 
to investigate the problem in the Australian context, with support through presenting 
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within the report. Donald Palmer, Principal Education Officer, Educational 
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information on the development of the Curriculum Materials Review Guide. 
It was only possible to establish the research bases for this report through extensive 
correspondence. P. Kenneth Komoski, Executive Director, Educational Products 
Information Exchange (EPIE) Institute, New York is to be dianked for supporting die 
conduct of the project and for providing information on die activities of the EPIE Institute 
at an earlier occasion. David L. Elliott, Director of EPIE Institute's Western Projects 
Office, Kenwood, California also provided helpful information at an earlier occasion on 
tile development of the instruments used by die EPIE Institute. Laurel Singleton, Senior 
Staff Associate, Social Science Education Consortium, Boulder, Colorado, is to be 
thanked for providing assistance in die conduct of die study. 

Robin Syme, Assistant Director of Cuiriculum and Cyndiia McArthur, Co-ordinator, 
Learning Resources, Ministry of Education, British Columbia are to be dianked, in 
particular, for offering information on die activities of die Canadian Exchange for 
Instructional Materials Analysis (CEIMA) in British Columbia and for editing die 
excerpt on die Canadian Exchange for Instructional Materials Analysis (CEIMA). Grace 
Parasuik, Director of Curriculum, Manitoba Education, offered information on die work 
of die CEIMA in Manitoba and also facilitated die project. 

The research also depended upon assistance provided by people in Europe and Asia. 
Ingvar Sigurgeirsson, Director of the National Centre for Educational Materials, 
Reykjavik, Iceland, offered information on a project to implement the Sussex Scheme in 
Iceland and also interviewed one of die Scheme's developers, Michael Sraut, Professor 
of Education, University of Sussex, England. Michael Eraut is to be dianked for editing 
die script of the interview and die text of die excerpt tided. Education Area, University 
of Sussex, appearing in the report. Wu Yongxing, Director, Curriculum and Teaching 
Materials Research Institute, Beijing, Peoples Republic of China, offered information 
on a project to apply the Sussex Scheme in China. 

Special credit is given to die audior's brother, Christopher Watt, who sacrificed his own 
time to assume sole responsibility for the technical side of die reproduction of the report, 
to negotiate rJie work load widi the typist, and to correct errors. Marjory Morris, is to be 
thanked for typing most of the report. Kerryn Langford, Lorraine Lovell, Pamela Curtain 
and Jennifer Walsh are tc be thanked for typing correspondence. Warren Brewer, 
Superintendent, Curriculum Development and Evaluation Section, Education 
Department of Tasmania, is to be thanked for providing die facilities to duplicate copies 
of the report. 

Permission to reprint excerpts of material reserved under copyright is granted by die 
following organisations: by die Education Department of Tasmania to publish quotations 
from die Resource Management Guide for Australian Schools, (c) 1983, from pages 
11,27,33 and 36, and die Learning Resources Review, from pages 8 and 28; by die Teachers 
College Press, Columbia University and die EPIE Institute to publish a quotation from 
The Educational Software Selector,{c) 1984, from pages 547-548; by the Association for 
Supervision and Curriculum Development, Alexandria, Virginia to publish a quotation 
from the article by P. Kenneth Komoski, 'Instructional Materials will not Improve until 
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we Change the System' from Educational Leadership, vol. 42, no. 7 on page 33; by Ingvar 
Sigurgeirsson to publish a quotation from the abstract of the thesis, Improving 
Curriculum Materials Development in Iceland through Curriculum Analysis; by the 
Australian Schools Catalogue Information Service to publish the content of a paper 
presented to Board Meeting no. 10 of the Australian Schools Catalogue Information 
Service and an excerpt from the minutes of the Board Meeting; and by Sage Publications 
to quote excerpts from the article by D. L. Madey and A. J. Stenner, 'Policy Implications 
Analysis: A Method for Improving Policy Research and Evaluation' from Improving 
Educational Evaluation Methods: Impact on Policy, edited by C. .B. Aslanian, on pages 32 
to 36. The excerpts published from Educational Technology : The Closing-in or the 
Opening-out of Curriculum and Instruction, by P. Kenneth Kombski, and A Look at 
Current State-wideText Adoption Procedures by Charles R. Duke are in the public domain. 
The map of Tasmania is published with the permission of the Australian Bureau of 
Statistics. 

Readers will note on page 74 that the author proposes an Addendum will be written to 
this report to analyse data remaining uncollected at the present time. Since completing 
the body of the report, the author hrs received notice from Mr. Komoski that it appears 
the questionnaire intended to collect this data had been mislaid. Because it is no longer 
feasible to collect this data, this document represents the final report. 
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CHAPTER 1 



THE INTRODUCnON 



1.1 Background 

The collection, the synthesis and the dissemination of qualitative information on 
instructional materials to teachers has represented an intractable problem in most 
contexts of the educational setting. The few, successful programs in education provide 
the evidence to substantiate this claim. The purpose of this report is to offer a planned 
solution to this problem for Australian education by basing that solution upon such 
estimable and successful programs operating in foreign contexts. 
The difficulty for an educational organisation to effect a research and development 
program results, in part, from several major problems inherent in the development, the 
analysis, the selection and the implementation of instructional materials. Klein (1978) 
has identified five major problems: determining the characteristics of quality in 
instructional materials; determining the types of research methodologies to provide 
information on instructional materials; incorporating learner-based verification and 
revision; defining responsibility for learning resulting from instructional materials; and 
specifying the rights of interest groups to determine the materials to be used. Each of 
these issues is now discussed. 

Although criteria applied by selectors and analysts are based upon a premise that they 
relate to the quality of instructional materials, there is little research evidence available 
to indicate that such criteria are sufficiently comprehensive for selectors and analysts to 
judge the intrinsic nature of quality in instructional materials. Research evidence is 
necessary to determine what makes for instructional materials of high quality. Criteria 
forjudging a range of materials can then be derived from the results of this research. 
A further difficulty is imposed by the limitations of research methodologies used to 
investigate instructional materials. Generally, experimental designs applying a 
pretest-teach-posttest model have been used to investigate what has been learnt from 
materials. Although these designs appear to be effective in providing answers about 
materials meant to develop cognitive behaviours, it is unlikely that such designs are 
appropriate for providing answers about materials meant to develop affective, 
psychomotor or a combination of behaviours. It is more likely that systematic 
observations and interview techniques, rather than experimental designs, will provide 
valid answers about how materials affect learners* values and physical performances. 
Furthermore, experimental designs cannot account for all the antecedent conditions and 
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contextual variabK^s that are involved in learning from instructional materials. It is 
evident that research methodologies to investigate various aspects of instructional 
materials need to be extended, if valid answers are to be provided. 
Learner-based verification and revision of instructional materials refer to procedures for 
gathering and analysing data obtained from field research of materials with appropriate 
groups of learners. Guidelines for learner-based verification and revision, however, have 
not overcome certain obdurate features. For instance, limits have not been clearly 
established as to which materials learner-based verification and revision should apply to; 
sampling procedures iiave not been clearly defined; issues relating to the replication of 
procedures and the generalisation of findings have not been solved; and the relationship 
between learner-based verification and revision and quality of instructional materials has 
not been established by research evidence. 

An additional difficulty is whether the publishers of instructional materials or the teachers 
who use them are to accept responsibility for the learning resulting from the instructional 
materials. Publishers have usually been criticised for any aspects of bias that might be 
present in instructional materials. On the other hand, publishers have sought to shift to 
classroom teachers responsibility for learning resulting from instructional materials. 
Although shared between publishers and teachers, there is a reluctance for responsibility 
to be accepted by either party. 

The final problem relates to the involvement of different interest groups in the selection 
of instructional materials. The influences of agencies funding instructional materials to 
maintain control of the materials' contents, the responsibilities of education departments, 
the activities of professional groups and the controversial assertion of rights by 
community groups to censor certain materials, have collectively tended to widen the 
numbers and composition of groups involved in the process of selection. 
These issues have not been addressed systematically in the Australian educational 
context. The prospective program proposed by the author is an attempt to correct such 
failures in the context of Australian education through an institutional project, partly 
research and partly service. 

1.2 The Rationale 

1.2.1 The Rationale Statement 

The purpose of the evaluation project is to present a plan for a program to establish a 
clearinghouse for exchanging qualitative and quantitative data on instructional materials 
used in Australian schools. The r. learcher investigates a range of elements necessary to 
plan a program suited to Australian requirements and conditions: its mission, capability 
and setting; its management; its personnel; its co-ordination and balance; its research 
activities; its development activities; its potential for relationships with ether 
institutions, agencies and organisations; its schedule; its facilities; its funding; and its 
product dissemination. 

The evaluation project, therefore, is a feasibility study through which the researcher 
intends to determine the optimal model for a prospective program to meet Australian 
requirements and conditions. By necessity, such a program plan is the consequence of 
an eclectic approach. The researcher proposes a program plan that embodies innovative 
methods, techniques and practices derived from a variety of foreign contexts. As well as 
determining which of these methods, techniques and practices can be adapted to suit 



Australian requirements and conditions, the researcher also probes the potential 
contributions by educational organisations in foreign contexts. 
The prospective program is based upon an assumption that the existing program to 
collect, to synthesise and to disseminate qualitative data on instructional materials in the 
context of Australian education has failed to recognise and to draw upon exemplar^ 
methods, techniques and practices used in foreign contexts. Furthermore, the program 
plan is based upon a belief that the existing program presents a deficiency of such a 
pervasive form that it cannot be remedied through adjustment of the existing program, 
but instead must be redesigned or replaced in toto. This belief is supported by 
observations that, firstly, there has been neither planned nor substantial investment in 
research and developmert activities at a national level in Australian education to analyse 
data on instructional materials, and secondly, those activities conducted at a state level 
in Australian education apply inadequate methods, techniques and practices to the 
problems of collecting, synthesising and disseminating data on instructional materials. 
The prospective program, therefore, is likely to redress several significant problems 
which Australian education has failed to provide solutions in the past. Firstly, the 
prospective program has the potential to redress perceived incoordination and 
imbalances between the missions of various Australian educational organisations at both 
national and state levels through rationalising the provision of services by redesigning or 
replacing those existing, but ineffective, services. Secondly, the prospective program will 
remedy the inadequate training of personnel through an inservice program. Thirdly, the 
prospective program will increase research and development capabilities through 
support from existing research and development activities at foreign organisations. 
Fourthly, the prospective program will replace ad hoc approaches by a planned schedule. 
Fifthly, the prospective program will extend the facilities of the participating Australian 
organisations. Sixthly, the prospective program will increase cost effectiveness of the 
work through each of these actions, in spite of its labour intensive traits. Lastly, the 
prospective program will disseminate suitable products intended to match instructional 
materials and educational programs, thereby enhancing student achievement. 

1.2.2 Background to Development of the Rationale 

The rationale for the evaluation project arose from correspondence between the 
researcher and P. Kenneth Komoski, Executive Director, Educational Products 
Information Exchange (EPIE) Institute, during 1986. On 20 August 1986, the researcher 
presented the following question to Komoski. 

"Would EPIE Institute be prepared to extend Its services or assistance to Australia, especial- 
ly with the view to establishing a national service, including computerise J databases, for selec- 
tion and evaluation of educational materials in Australia?" 
Komoski's response of 5 September 1986 to this proposal was favourable. 

"If an appropriate Australian educational institution wished to work with EPIE to extend its 
services to serve the needs of Australian schools, EPIE would be very mterested in exploring 
the means through which this could be done." 

The indication contained in this correspondence to establish a working relationship 
between the EPIE Institule and an Australian educational organisation remained 
dormant for some months. Then in December 1986, the researcher discussed the nature 
of his research on instructional materials, and his contact with the EPIE Institute with 
Glenn C. Pullen, the Tasmanian representative on the Board of Directors of the 
Australian Schools Catalogue Information Service (ASCIS). 



Almost immediately, Pullen presented this information to several of his colleagues on 
the ASCIS Board of Directors. Acting upon this information, the ASCIS Executive 
Director, Georgina Cane, undertook a preliminary contact with the EPIE Institute. 
In the meantime, the researcher had negotiated with Phillip W. Hughes, Professor of 
Education, University of Tasmania, to conduct an evaluation project on the topic as part 
of the requirement for a Master of Education degree. Following successful negotiations 
with Hughes, the researcher visited both the Australian Council for Educational 
Research (ACER) at Hawthorn, Victoria, and the ASCIS at Camberwell, Victoria, during 
August 1987. The purpose of these visits was to discuss with staff members the prospects 
of either of these organisations becoming involved in a co-operative venture with the 
EPIE Institute. The discussion with the Executive Director of the ASCIS was particularly 
positive. 

This led the researcher to commence planning the evaluation project. Following 
discussions with colleagues at workshops conducted by Hughes, an evaluation design 
based upon the Context^Input-Process-Product (CIPP) Model, proposed by Stufflebeam 
et al. (1971b), was adopted. The collection of information for the evaluation project and 
the writing of the report commenced soon afterwards. 

Meanwhile, Pullen presented a formal proposal to the ASCIS Board of Directors on 7 
October 1987. "^e proposal supported the evaluation project being undertaken by the 
researcher. Furti ^^rmore,- an initial milestone was then set at the presentation of a 
submission to the Curriculum Development Centre (CDC) by the researcher and Pullen, 
following the meeting of the ASCIS Board of Directors during April 1988. 
Late in 1987, Pullen and the researcher agreed that a submission should be prepared for 
presentation to the forthcoming ASCIS Board meeting during April 1988. During the 
early months of 1988, the researcher presented Pullen-with the chapters of the project 
report that constitute the context evaluation. From this information, Pullen developed 
a submission to be placed on the agenda for the ASCIS Board meeting, following 
determination of a consensus with the researcher on its contents. 
The submission was received positively when the ASCIS Board met on 13-14 April, 1988. 
During the course of the meeting, Pullen reported that he was able to discuss the 
implications for action in the submission with the Director of the Curriculum 
Development Centre, who is also a representative on the ASCIS Board. The ASCIS 
Board agreed that the submission would be presented to the Curriculum Development 
Centre with one amendment. The submission document, which includes the 
amendment, is reproduced as Appendix A, 

On the 26th August 1988, the researcher met for an hour with Brent Corish, the Director 
of the Curriculum Development Centre at Canberra, A.C.T. The purpose of the meeting 
was to discuss the intent of the project. The meeting commenced with discussions on 
the background to the project and the findings that could already be determined. The 
researcher was able to contrast the relative effectiveness of the approaches currently 
applied in Australian education to select and to evaluate instructional materials with 
those approaches applied by the Educational Products Information Exchange (EPIE) 
Institute in the United States and the Canadian Exchange for Instructional Materials 
(CEIMA.) in Canada. At this point, Corish was able to inform the researcher of current 
developments in planning for the introduction of a national curriculum policy and 
guidelines for Australian schools. Both Corish and the researcher discussed the 
implications of the proposal to develop a clearinghouse to exchange information on 
instructional materials for the intended national curriculum. They agreed that these 
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developments would be compatible and particularly desirable outcomes. The researcher 
was able to pass on to Corish copies of correspondence from the EPIE Institute that 
related to interest expressed by its Executive Director for that organisation to become 
involved in Australian education . Corish indicated that an intention of the Curriculum 
Development Centre is to organise a conference or workshop on approaches to select 
and evaluate instructional materials, but he queried the necessity to invite personnel from 
foreign educational organisations because it would represent an additional expense. 
Furthermore, Corish felt that there would be no need to involve foreign educational 
organisations in planning and stmcturing the activities of a clearinghouse for Australian 
education unless it could be demonstrated that the practices, techniques and methods 
applied by those organisations were of such a nature as could not be applied by Australian 
educators. The researcher was able to reply that he believed such an assessment could 
not be made until the final report of the study was considered. Corish acceded, and he 
requested that a copy of the final report be made available to him before the ASCIS Board 
meeting planned for October 1988. In a concluding statement, Corish said that he 
believed planning of the clearinghouse will commence during 1989. 

1 .3 The Objectives 

This evaluation project has three main objectives: to explicate the objectives of a 
prospective program to collect, to synthesise and to disseminate qualitative and 
quantitative data on instructional materials to Australian schools; to investigate the 
potential adaptation of innovative methods, techniques and practices to analyse 
qualitative and quantitative data on instructional materials; and to determine the 
resources that both Australian. and foreign educational organisations may contribute to 
planning, structuring, implementing and recycling the procedures of the prospective 
program. 

The context evaluation in Chapter 2 and Chapter 3 contains extensive treatment of the 
first objective. The explication of objectives for the prospective program is based upon 
a comparison of the findings that resulted from context evaluations of the existing systems 
for both research and service in American and Australian education. 
The second objective is treated extensively in both the context evaluation and the input 
evaluation. In the former case, treatment of this objective encompasses examination of 
the methods, the techniques and the practices applied to analyse qualitative and 
quantintive data on instructional materials within the programs of three foreign 
educational institutions, two in the United States of America and one in Canada. In the 
latter case, the historical development of the applications of these methods, techniques 
and practices is traced within a range of foreign contexts, giving particular reference to 
adaptations undertaken in geographical contexts aUen to the site of original development. 
The input evaluation in Chapter 5 contains extensive treatment of the third objective. 
The planned change model is applied to assess the relevant capabiUties of Australian ' 
educational agencies to provide resources for the prospective program. A questionnaire, 
which employs a technique termed Policy Implications Analysis, is administered to a 
panel of policy makers in foreign educational organisations to quantify their attitudes 
towards strategies that their agencies may contribute to the prospective program. By 
determining a congruence between these two assessments, a design for implementing the 
prospective program is specified on criteria of institutional capability, program elements, 
management, personnel, funding, schedule, facilities, and communication. 
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The products of the evaluation project comprise three documents: the report on the 
evaluation project; a review on the treatment of the topic in national reports on education 
emanating from the United States of America, and a supplementary report on existing 
practices used to analyse data on instructional materials in Tasmanian education. The 
report of the evaluation project is contained within this document, whilst the other two 
products are subsidiary documents. 

The selection of component documents in the evaluation project was determined by the 
requirements of the post-graduate course to which the documents were to be submitted. 
The formats of each of these documents, however, were refined later to meet the 
requirements of specific audiences to which the documents are directed. There were two 
main audiences to which these documents were written: firstly, decision-makers and 
administrators at a national level in Australian, American and Canadian educational 
systems; and secondly, a range of personnel at the state level education system in 
Tasmania. Both the report of the evaluation project and the review of national reports 
on education were directed towards the former audience, whilst the supplementary 
report on existing practices in Tasmanian education was directed to the latter audience. 

1.4 The Evaluation Design 

1.4.1 Research Plan 

A major problem confronting the researcher concerned the selection of an approach to 
educational evaluation that is pertinent to the programs of educational research and 
development agencies. It was soon recognised that most approaches to educational 
evaluation pertain to school programs, and there were few existent approaches available 
for the evaluation of research and development agencies. In view of this finding, the 
researcher decided to review texts that presented basic syntheses of approaches to 
educational evaluation in order to select an appropriate research model for the evaluation 
design. This led the researcher to consider two types of evaluation models: designs that 
are specifically appropriate to the evaluation of the programs of research and 
development institutions; and designs that are appropriate to the evaluation of school 
programs. 

A set of criteria, expounded by Worthen and Sanders (1973: 210-215), was used to judge 
each of the approaches to educational evaluation reviewed. Important components of 
educational evaluation are represented by these twelve criteria: 1. definition; 2. purpose; 
3. key emphasis; 4. role of evaluator; 5. relationship to objectives; 6. relationship to 
decision-making; 7. types of evaluation; 8. constructs proposed; 9. criteria for judging 
evaluation; 10. implications for design; 11. contributions; and 12. limitations. 
As a result of this review, the Context-Input-Process-Product (CIPP) Model proposed by 
Stufflebeam et al. (1971b), was selected because it matched best the characteristics of the 
problem to be researched. The integrity of the match between the CIPP Model and the 
characteristics of the research problem is described for eacli criterion. 

1. The problem defined in the project matches this characteristic in the CIPP Model; it 
pertains to defining, obtaining and using information for decision-making. 

2. The purpose for the evaluation design of the project matches the purpose of the CIPP 
Model; it pertains to providing relevant information to decision-makers. 

3. The key emphasis of the project matches this characteristic in the CiPP Model; the 
key emphasis is to provide an evaluation report for decision-making. 
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4. The role of the evaluaior in the project matches the role defined in the CIPP Model; 
it pertains to providing specialist, evaluative information to decision-makers. 

5. The terminal outcomes of both the context evaluation and the input evaluation in the 
project match the outcomes of these evaluation types in the CIPP Model; the outcome 
of the context evaluation is.Jo set the objectives and other characteristics of the 
prospective program, whilst the outcome of the input evaluation is to determine ways 
educational organisations in foreign contexts may assist to provide the means to reach 
these objectives. 

6. The relationship of the project to decision-making matches the definition of this 
relationship in the CIPP Model; it provides information for use in making decisions on 
the objectives of the prospective program and on the extent of involvement of foreign 
organisations* 

7. Two types of evaluation inherent in the CIPP Model are applied in the project: a 
context evaluation in the contingency mode; and an input evaluation for neomobilistic 
change. 

8. The constructs for each type of evaluation applied in the project comply with the 
constructs proposed in the CIPP Model; the context evaluation is used in the project to 
plan decisions; and the input evaluation is used to program specific decisions on the 
nature of involvement of foreign organisations. 

9. Criteria used in the project to judge evaluation comply with criteria specified in the 
CIPP Model; internal validity, external validity, reliability, relevance, importance, scope, 
credibility, timeliness, pervasiveness and efficiency are criteria applied in the evaluation 
project. 

10. The implications for design of the project match the implications for design in the 
CIPP Model; the evaluator does not apply experimental design, but uses a systems 
approach. 

11. The contributions of the evaluation project match the contributions that are provided 
by the CIPP Model; the evaluation project contributes a service function to 
decision-makers of Australian and foreign educational organisations, and the evaluator 
takes account of feedback from decision-makers of these organisations. 

12. The evaluation project would seem to be governed by limitations similar to those 
affecting the CIPP Model; value concerns are not emphasised, the decision-making 
process and methodology are not defined, all activities in the project are not clearly 
evaluative, although an attempt is made to overcome an inherent limitation of cost and 
complexity by restricting the evaluation to the context type and particular aspects of the 
input type. The statement of the rationale implies clear limitations on the use of the types 
of evaluation within CIPP Model. As the project is essentially a feasibility study, only the 
context and input types are appropriate to the research plan. 

Stufflebeam et al. (1971b) define the purpose of context evaluation as serving planning 
decisions in order to determine objectives. Two modes of context evaluation are 
identifiable: contingency and congruence. Because the contingency mode involves 
identifying opportunities and forces beyond the boundaries of the immediate system to 
promote improvement in it, characteristic techniques of this type are employed to probe 
external forces and to predict projections into the future. 

The use of one technique, however, does require particular comment. A decision was 
made to collect baseline data at a macro-level from within the immediate system, a 
technique more characteristic of conguence evaluation when the concern is with 
accounting for discrepancies between objectives and their terminal outcomes. 



ERIC 



Stufflebeam et al. (1971b) have commented upon the doubtful value of collecting 
baseline data at a macro-level for contingency evaluations, for several reasons: firstly, 
the gain from baseline data at a macro-level is unclear; secondly, greater inputs of data 
are required at a macro-level; and thirdly, the nature of data required for prognostic 
purposes is different from data required to measure achievement levels. The limitations 
of collecting baseline data in the Tasmanian context are apparent. Although this 
collection served a need for such data at a state level, it failed to address the requirements 
of a national survey. To some extent, this problem was overcome by the collection of data 
at a national level but, because these data were non-random, the results may be 
unrepresentative. 

The CIPP Model, however, proved to be inadequate to explicate the objectives and other 
characteristics of the prospective program. During the search, the researcher identified 
two examples of designs to evaluate programs of research and development agencies: a 
design proposed by Scriven et al. (1971); and a design proposed by Stufflebeam et al. 
(1971a). This latter example was selected to provide a statement of objectives and other 
characteristics of the prospective program because of its perceived compatibility to the 
CIPP Model. 

Stufflebeam et'al. (1971b) define the purpose of- input evaluation as providing 
information for determining how to use resources to meet program goals. This is 
accomplished through identifying and assessing three characteristics: firstly, the relevant 
capabilities of a responsible agency; secondly, strategies for achieving program goals; and 
thirdly, designs for implementing a selected strategy. In this study, such a process 
assumes a particular form. In the first instance, the relevant capabilities of the two 
Australian educational agencies are identified and assessed, in the second instance, a 
technique is administered to probe strategies that foreign educational agencies may apply 
to achieve program goals, and in the third instance, the design for implementing the 
program is determined through establishing a congruence between existing capabilities 
of Australian educational agencies and strategies that foreign educational agencies can 
provide. 

The CIPP Model ascribes a variety of methodologies for input evaluation, depending 
upon whether large or small change is involved and whether high or low information grasp 
is available to support the change. The conceptualisation of the decision-making 
process, described by Stufflebeam et al. (1971b), is used to determine that a neomobilistic 
decision setting pertains to this problem. The advanced attributes of methods, 
techniques and practices used in foreign contexts determine that innovative activity for 
inventing, testing and diffusing new solutions should be applied. Such a large change, 
however, is supported by little information within the Australian context. The planned 
change model, recommended by Stufflebeam et al. (1971b) for neomobilistic settings, is 
applied within this study in two ways: firstly, to assess the capabilities of the releva.i 
Australian educational organisations; and secondly, to specify the design for the 
prospective program, in which innovative methods, techniques and practices derived 
from foreign contexts are superimposed upon existing practices in Australian education. 

1.4.2 Data collection 

information required to answer questions on the topic was identified through searches 
oi^ information databases on education and through personal communications. A 



O 8 

ERJC J 8 



considerable proportion of the fuiormation required to conduct the evaluation project 
had been obtained beforehand. 

The databases, Resources in Education and Current Index to Journals in Education^ 
compiled by the Educational Resources Information Center, were the main sources for 
collecting information. Information was gathered by correspondence from state 
departments of education, the Australian Council for Educational Research (ACER) and 
the Australian Schools Catalogue Information Service (ASQS) in Australia; from the 
Educational Products Information Exchange (EPIE) Institute and the Social Science 
Education Consortium (SSEC) in the United States, from the Canadian Exchange for 
Instructional Materials Analysis (CEIMA) in Canada, from Mr. Ingvar Sigurgeirsson in 
Iceland,and Dr. Michael Eraut in England.. 

1.4.3 Data Collection Schedule 

Procedures for collecting data are described in the relevant parts of the report. Data 
were collected during the conduct of both the context evaluation and the input evaluation. 
For the context evaluation conducted in both Tasmania and other Australian states, most 
data were collected between October 1987 and December 1987. For the input 
evaluation, most data were collected between April 1988 and August 1988. 

1.4.4 Techniques for data analysis 

Qualitative data only were analysed during both the context evaluation and the input 
evaluation. Therefore, statistical methods were not applied to analyse data. 
Qualitative data formed the bases forjudging quality. Such data were used to make two 
types of judgements: firstly, on relative levels of performance in different contexts of the 
educational setting; and secondly, on predicting projections. In both cases, the use of 
qualitative data meant that criteria were non-numerical. In the former case, criteria were 
based upon categorical levels used to measure observations. The standards for judging 
quality were based upon the researcher's knowledge of the topic. 

1.4.5 Reporting results 

Qualitative data were reported through non-numerical summaries or descriptive 
analyses. Data from both the multisite case study in Tasmania and the policy implications 
analysis were reported through non-numerical summaries, whilst information on 
documents was reported by descriptive analysis. 

1.4.6 Constraints upon the evaluation 

The major constraints that faced the researcher concerned both available time and 
funding. The researcher was required to balance the development and the 
implementation of the research plan with the requirements to collect information 
frequently from foreign sources, and the need to present the report, in part or in whole, 
to various audiences. The balance was maintained through the researcher's efforts at 
both planning and collecting a large proportion of the information prior to commencing 
the writing of the report. 

Although a proposal for a small sum to fund the project was presented to the ASCIS, this 
proposal was unsuccessful. The evaluation project was funded entirely by the researcher. 
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This funding was spent on searches of information databases and inter-library loans, for 
payments to a typist and for mailing correspondence. 
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CHAPTER 2. 



THE AUSTRALIAN CONTEXT 



In this chapter, the topic is examined in the Australian context. This is accomplished in 
two ways. Firstly, the programs of national educational agencies that provide information 
on instructional materials are outlined through a descriptive account. Secondly, both a 
survey of documentary evidence and a multi-site case study were conducted in the 
Tasmanian context to provide data on the extent to which techniques for providing 
information on instructional materials are being applied in the Australian context. In this 
study, the staffs of service agencies of the Education Department of Tasmania were 
surveyed to determine the characteristics of the procedures they use to evaluate 
educational products. The results of this study were extrapolated to the wider Australian 
context through an examination of a sample of documents from other state education 
departments. 

2.1 Agencies and Programs 

The first, and only, proposal to develop a national system of evaluations of curriculum 
materials was presented by The Australian Council for Educational Research (ACER). 
This endeavour, however, was aborted during the 1970's because this activity was to be 
encompassed within the mission of the newly founded Curriculum Development Centre 
(CDC). In spite of several projects fostered by the Curriculum Development Centre to 
provide evaluations of curriculum materials within specific curriculum areas, the 
Curriculum Development Centre has not attempted to develop a national system for the 
evaluation of a comprehensive range of educational products. The foundation of the 
Australian Schools Catalogue Information Service (ASCIS), however, has now made the 
development of a national system for evaluations of instructional materials a realistic 
possibility. 

State education departments in Australia, moreover, have contributed to the 
development of evaluations of instructional materials. However, these efforts have been 
conducted independently for the most part. There has been no attempt, with the 
exception of the National Software Co-ordination Unit, to co-ordinate these activities. 
It is within this climate that the contribution to the field by national educational agencies 
in Australia is now discussed in detail. 
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2.1. 1 The Australian Council for Educational Research! (ACER) 

The Australian Counc/] for Educational Research (ACER) was founded in 1930 as a 
national institute for educational research. In a large part, this educational research has 
taken place in test development. Funding is derived from the sale of publications, in 
particular test instruments, and from grants for educational research funded by 
governmental sources. Tlie ACER is located at Hawthorn, Victoria, Austialia. 

2.1.1.1 The Curriculum Materials Review Guide 

The Australian Council ior EducationaLResearch (1976) has developed an instrument 
to evaluate curriculum materials. Its developer, Jeffery (1987) reported that: 

"I made a detailed study of the EPIE Institute procedures and materials about ten years ago 
when I was setting up the curriculum materials reviewing system for the ACER. At that time 
it was envisaged that the ACER might play a wider role m the evaluation of curriculum 
materials on a national scale. However, the advent of the CDC changed all that and we have 
only maintained our re-/iew svstem quietly, mostlj for the materials which we distribute. 
The original Curriculum Materials Review Guide owes much to the woric of EPIE but also the 
woric that I and colleagues did at the University of Papua New Guinea Teaching Materials and 
Resource Centre. I also worked closely with ACER staff members to create an instrument 
that would woric for the sort of materials in which we are mterested and with the sort of profes- 
sional people we were dealing with. The original introduction to the Curriculum Materials 
Review Guide fully aclcnowledged the sources". 
The third edition of the Curriculum Materials Review Guide consists of seven sections: 1.0 
Review Identification; 2.0 Purpose and Basis; 3.0 Teacher's Guide; 4.0 Scope and 
Sequence; 5.0 Methodology; 6.0 Student Evaluation; and 7.0 Overview. This 
instrument employs Tyler's objectives model of curriculum development and includes 
both descriptive-analytic and evaluative; functions but omits a decision-making function. 
Despite the inaction of the Curriculum Development Centre in applying techniques to 
evaluate curriculum materials, it seems unlikely that The Australian Council for 
Educational Research will endeavour again to apply the Curriculum Materials Review 
Guide to the evaluation of curriculum materials nationally. 

2.1 .2 Curriculum Development Centre (CDC) 

The Curriculum Development Centre was established on 13 June 1973 as a result of a 
decision by the federal Australian government. Legislation in the form of the Curriculum 
Development Centre Act, 1975, formally instituting the Curriculum Development Centre, 
was passed by the Federal Australian Parliament on 27 May 1975. 
As a result, the Curriculum Development Centre was allocated th(j functions of devising 
and developing school curricula and curriculum materials, undertaking, promoting and 
assisting in research into matters related to school curricula and curriculum materials, 
collecting, assessing and disseminating information relating to school curricula and 
curriculum materials, and' arranging for the printing and publication of school curricula 
and curriculum materials. 

These operational functions of the Curriculum Development Centre, however, have not 
been performed without constraint. Du/ing the early 1980's, the Curriculum 
Development Centre was deactivated because of funding restrictions. Following 
enactment of legislation, the Curriculum Development Centre was reconstituted during 
1983. In July 1984, the new Curriculum Development Centre took the form of one of 
four divisions of the Commonwealth Schools'Commission. This arrangement continued 
until 30 November 1987 when the staff of the Commonwealth Schools Commission was 
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transferred to the newly formed Department of Employment, Education an- Training. 
The Curriculum Development Centre continues to operate in the Department of 
Employment, Education and Training reporting through a Curriculum Development 
Advisory Committee. 



2.1.2.1 The Projects 

The program of the Curriculum Development .Centre has comprised three types: funded 
projects; conferences; and an information network. The projects, which are funded for 
specific periods, fall into three major categories: development projects; seed projects; 
and research projects. The development projects are directed towards either producing 
instructional materials or providing approaches to classroom transactions. Seed projects 
are feasibility studies for larger-scale projects. Research projects are conducted to 
provide priorities for future work in curriculum development in Australia. 
The process for establishing projects has changed since the foundation of the new 
Curriculum Development Centre. Any person or agency has been able to present a 
submission, which has then undergone a process of testing and consultation. An 
important forum for testing and consultation is the regular meetings of the directors of 
curriculum from the education departments of all states and territories. 

2.1.2.1.1 The National Software Co-ordination Unit (NSCU) 

One project, the foundation and the development of the National Software Co-ordination 
Unit (NSCU), is now described in detail because of its close relationship to the 
prospective program proposed by the author in this document. 
The National Software Co-ordination Unit (NSCU) was founded by the Curriculum 
Development Centre during January 1987, following a feasibility study (Western 
Australian Educational Computing Consortium, 1986). The aims of the National 
Software Co-ordination Unit are to provide information on computer courseware and 
computer-related instructional materials nationally, and to co-ordinate the exchange of 
information on the development of computer courseware. Four developments have 
occurred during 1987 to facilitate these aims: the establishment of a Software 
Co-ordinators Advisory Group; the establishment of a National Workshop on Software 
for Education; the formation of a network for sharing information on computer education 
projects; and the development of a database of computer courseware evaluations. 
The Software Co-ordinators Advisory Group has been formed to improve 
communication between educational systems on matters of courseware development and 
evaluation. The National Workshop on Software for Education has been established to 
communicate information on computer courseware between producers, education 
systems and schools. The network is intended to facilitate the exchange of information 
between education systems on their courseware development projects. The database of 
computer courseware evaluations will facilitate the organisation on a national basis of 
evaluations of computer courseware already provided by state computer centres. The 
evaluations will be entered into the national database of the Australian Schools Catalogue 
Information Service (ASCIS). Each evaluation will be annotated on a set of descriptors 
providing both background information (a description of the package, a list of the 
computer hardware and software required to use the package, the relevant curriculum 
areas, suitable grade levels, references to published evaluations and case studies) and a 



13 



descriptive review of the usefulness of the courseware package in the classroom. It is 
intended that evaluations on the. database will be available to teachers early in 1988. 

2.1.3 The Australian Schools Catalogue Information Service (ASCIS) 

The Australian Schools Catalogue Information Service (ASCIS) was formed because the 
staffs of many educational organisations expressed the need for a national cataloguing 
service for Australian schools. A Planning Group was established during 1981 to direct 
the formative development of ASCIS. On 27 August 1984, ASCIS was incorporated as 
a company limited by guarantee. The members of ASCIS are the Ministers responsible 
for education in the States and Territories, the Minister responsible for the 
Commonwealth Schools Commission, the Executive Secretary of the National Catholic 
Education Commission and the Chairman of the National Council of Independent 
Schools. Each member of the company, in turn, appoints one member to the Board of 
Directors. The Board is responsible for formulating policy, effecting accounting, 
recommending the budget and determining developmental activities. 
Late in 1983, a staff was appointed to form an ASCIS secretariat, which at present consists 
of an executive director, a librarian, two cataloguers, an office manager, and two clerical 
assistants. The ASCIS secretariat is based in Melbourne, Victoria. A Technical Matters 
Sub-Committee has been appointed by the Board to advise on technical matters relating 
to the services provided by ASCIS. 

Early in 1984, ASCIS contracted ACI Computer Services to develop the computer 
database. ACI Computer Services provided an adaptation of the Dortmund 
Bibliographic Information System and the Leuven Integrated Bibliographic System to 
develop on-line facilities to catalogue, to search, to provide information on serials, to 
compile information on circulations to users, to provide statistical information for 
managerial requirements, and to allow acquisition of amendments to orders. In order 
that records can be processed from a variety of sources, ASCIS commissioned a 
consultant. Manning, to propose a recognised set of standards for formatting catalogue 
records. As a result of the consultant's report (Australian Schools Catalogue Information 
Service, 1985a), AUSMARC III was selected as the recognised standard upon which the 
cataloguing sub-system of the database is now based. This standard provides that 
descriptors, as follows, are specified for cataloguing and searching: name; title; general 
material designation; statement of responsibility; edition; publication information; 
collation statement; series; local accession number; general system number (ASCIS 
order number); ISBN; call number; location information; subjects; and general notes. 
Consequently, the Australian Schools Catalogue Information Service (1985b) published 
a guide for teacher-librarians to catalogue instructional materials in accordance with 
ASCIS standards. The list, compiled by two project officers, was field-tested in a sample 
of schools and state cataloguing agencies across the nation during 1983. The list was 
revised on the basis of this field-test, and subsequently published. The list, which uses 
Australian English terminologies, is designed so that it can be used by school students. 
In addition to the catalogue record, which provides bibliographic information, in the form 
of an annotated listing, there is provision for 'value-added' information to be added to 
each record. It is envisaged by the ASCIS staff that such 'value-added' information would 
consist of descriptive analytical and evaluative reviews, but could be used instead for 
evaluations based upon instructional design analysis. To date, the only 'value-added' 
information added to the database is based upon descriptive analytical reviews provided 
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in Scan, the journal of the Library Services, New South Wales Department of Education 
and the New South Wales Curriculum Information Network. At present, the Australian 
Schools Catalogue Information Service (1987) is developing a set of standards for 
'value-added' information. The current edition requires that 'value-added' information 
be specified to contain a 'header' and a summary description, although provision is being 
made for specific subjects of the database, such as those being developed as a result of 
projects funded by the Curriculum Development Centre: the Australian Curriculum 
Information Network (ACIN); and the National Sofuvare Co-ordination Unit (NSCU). 
This set of standards requires that the format includes specific presentations for the 
descriptors of the abstract 'header', summary description, audience, grade level and 
availability information. 

2.1.3.1 Services 

2.1.3.1.1 Outreach 

During an operational period of three years, ASCIS has extended services to 
approximately 4.000 of Australia's 10,000 schools. A further 2,000 schools receive 
catalogue records delivered from existing cataloguing systems of state education 
departments. 

During this period, a survey of staffs of some 800 Australian school libraries was 
conducted by ASCIS. From the staffs of 323 schools that responded, the data from 300 
returns, which represented both state and independent schools, were analysed. The aims 
of the survey were to determine how the ASCIS users are using ASCIS services, what 
sorts of results they are obtaining and v/hat are the effects of these results. For both users 
and non-users, ASCIS wished to determine how they saw the future in relation to ASCIS 
services. The results of the survey have been published (Australian Schools Catalogue 
Information Service, n.d.). 

2.1.3.1.2 Catalogue Services 

At present, catalogue services are available to meet the needs of a diverse group of users. 
These are available in five forms: magnetic tape; catalogue cards; floppy disks; computer 
output microfiche; and dial-up access. Magnetic tape output allows users with their own 
automated services to load the entire ASCIS database onto their own systems. Catalogue 
cards allow users to order sets of catalogue cards for selected records. 

2.2 Empirical Approaclies to Evaluate the Australian Context 

In order to verify this evidence, the researcher decided to apply empirical methods to 
gather qualitative data on the application of techniques used by Australian educators. 
Two designs were used to gather this information: a multi-site case study conducted in 
Tasmania; and documentary evidence collected from state education departments in 
Australia. 

2.2.1 The Case Study in the Tasmanian Context 

It was decided to investigate the Tasmanian context for three reasons: firstly, because it 
was believed that the results from an empirically-based survey would be replicable in a 
national setting; secondly, because it was found to be impracticable to gather 
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representative data on a national basis; and thirdly, because a survey of the Tasmanian 
context would provide valuable information for a rsview of such practices at a state level. 

2.2.1.1 The Research Problem 

In The Review of Efficiency and Effectiveness of the Education Department (Government 
of Tasmania, 1982: 31-33), the consultant, Hughes, identified four phases in the 
development of organisational structures in the Education Department of Tasmania. This 
development has been concomitant with an expansion of educational services to teachers. 
During the course of this development, the staffs at several service agencies of the 
Education Department of Tasmania have developed and applied various techniques to 
evaluate instructional materials independently. 

As the staffs at different service agencies apply various techniques, that range in their 
reliability and validity, to report information to teachers on instructional materials, it was 
judged to be important to assess the characteristics of these techniques. It can be 
expected that assessments of the applications of a variety of techniques at differentservice 
agencies would mdicate information on both the extent of reliability attained and the 
characteristics inherent in the sequence for reporting qualitative data on instructional 
materials in an Australian setting. 

When there is extensive application of formal procedures by the staffs of service agencies, 
it can be exp'^cted that there would be less probability of random errors occurring in the 
measurement of consistency between observations. ITie extent to which these staff 
members use formal procedures, which provide more reliable scores, will also cause more 
valid qualitative information for matching curriculum materials to the instructional 
design to be reported. The extent of the validity of such information is also likely to be 
reflected in the stringency of adherence to the steps in an inlierent, generic sequence of 
techniques. 

The study sought to determine the following problems concerning the application of 
techniques to evaluate curriculum materials. Firstly, the prediction was made that there 
would be extensive application of informal procedures by the staffs of service agencies 
of the Education Department of Tasmania, to the extent that there would be lower 
reliability from measures than is necessary to reduce random errors to an acceptable level. 
Secondly, the prediction was made that there would be only an incomplete application 
at most service agencies of the set of techniques in the sequence, to the extent that there 
would be lower validity from measures than is necessary to reduce systematic errors to 
an acceptable level. 

2.2.1.2 The Method 

2.2.1.2.1 The Target Population 

The target population for the multi-site case study comprised the staffs of service agencies 
of the Education Department of Tasmania that are involved in using techniques to 
evaluate educational products. Since a directory of such service agencies is unavailable, 
they were identified by several means: from the researcher's kiiowledge; from the 
telephone directories for Tasmanian area codes published by Telecom Australia; from 
a comprehensive list of ancillary educational establishments provided by the Executive 
Support Services of the Education Department of Tasmania; and from a list of twenty 
service agencies selected for survey by the executive group responsible for the project, 
Learning Resources Review. 
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Three distinct types of service agencies were identified within the population: service 
centres; resource centres; and environmental centres. These service agencies of the 
Education Department of Tasmania are administered through three arrangements: a 
central administration controls the service centres; regional offices, located at Hobart in 
southern Tasmania, Launceston in,. northern Tasmania and Burnie in north-western 
Tasmania, each provides a range of resource centres to service schools within its region; 
and collaborative arrangements between the Education Department of Tasmania and 
other state governmental agencies, the Department of Lands, Parks and Wildlife and the 
Department of Sport and Recreation, administer the environmental centres. 
The criterion for including a service agency within the target population was its 
independence as an organisation. Service agencies in the northern and north-western 
regions of Tasmania administer a combination of services and resource centres. Each of 
these services and resource centres was not considered an independent agency so that, 
in these cases, only the service agency administering this complex was included in the 
population. On this basis, the population included in the survey comprised nineteen 
service agencies. Because of the small size of the population, it was unnecessary to survey 
a sample. 

Tne staffs of the seventeen service agencies, who responded to the survey, belonged to 
four types of organisations: service centres; resources centres; environmental centres; 
and independent authorities. These are listed below under each category. 
Service Centres 

Site 1: The Curriculum Development and Evaluation Section located at the site of the 
Division of Educational Programs, North Hobart, Tasmania. 

Site 2: The Media Library, Curriculum Resources Section located at the site of the 
Division of Educational Programs, North Hobart, Tasmania. 

Site 3: The Student Services Section located at the site of the Division of Educational 
Programs, North Hobart, Tasmania. 

Site 4: The CumGulum Ser.'ices Section located at the site of the Division of Technical 
and Further Education, Hobart. 

Site 5: The Elizabeth Computer Centre located at the site of the Elizabeth Matriculation 
College, Hobart, staffed by consultants in computer studies and catering to grade 1 
through to grade 12. 

Site 6: The Regional Resource Centre located at the site of the Launceston Teachers 
Centre, Launceston. 
Resource Centres 

Site 7: The Language Arts Resources Centre located at Hobart, staffed by consultants 
in the language arts, and catering to grade 1 through to grade 10. 
Site 8: The Bowen Road Resource Centre located at the Bowen Road Primary School, 
Moonah, a suburb of Hobart, staffed by consultants in early childhood and primary 
education and catering to grade K through to grade 6. 

Site 9: The Gifted and Talented Resource Centre located at the South Hobart Primary 
School, South Hobart, a suburb of Hobart, staffed by a consultant on student giftedness 
and talent and catering to grade 1 through to grade 6. 

Site 10: The Goodwood Arts Centre located at Goodwood Primary School, Moonah, 
staffed by consultants in the visual arts and the performing arts and catering to grade 1 
through to grade 6. 
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Site 11: The Flagstaff Mathematics Centre located at the Flagstaff Primary School, 
Warrane, a suburb of Hobart, staffed by consultants in mathematics and catering to grade 
1 through to grade 6. 

Site 12: The Maths Science Resource Centre located at the site of the Launceston 
Teachers Centre, Launceston, staffed by consultants on mathematics and science and 
catering to grade 1 through to grade 6. 
Environmental Centres 

Site 13: The Molesworth Environment Study Centre located at the Molesworth Primary 
School, Molesworth, a rural settlement 24 km north-west of Hobart, and operated for 
environmental studies in a lowland, dry sclerophyll forest environment. 
Site 14: The Sprent Environment Centre located at the Sprent Primary School, Sprent, 
a rural settlement 32 km south-west of Devonport, and operated for rural studies in farm 
organisation and for environmental studies in a wet sclerophyll forest environment. 
Site 15: The Woodbridge Marine Studies Centre located at Woodbridge, a coastal 
township depending upon orcharding and fishing, 37 km south of Hobart, and operated 
for environmental studies of a marine environment. 

Site 16: The Port Arthur Education Centre located at Port Arthur, the site of a 
mid-nineteenth century penitentiary for convicts transported from Britain, on the 
Tasman Peninsula 104 km south-east of Hobart, and operated for historical investigation 
of this site. 

Independent Authority 

Site 17: The Schools Board of Tasmania located at Sandy Bay, a suburb of Hobart, an 
accrediting agency which plays an important role in the adoption of instructional 
materials at the secondary level (grade 10 through to grade 12), 

2.2.1.2.2 The Measurement Instrument 

Since the researcher was unable to identify a measurement instrument that could be 
administered to assess the characteristics of the various techniques used to evaluate 
instruciional materials, he decided to develop his own instrument. It was recognised that 
Biickell and Aslanian (1979) had identified a set of slages - pilot-trial, pilot-test, 
field-trial and field-test - characteristic of the learner-based verification and revision 
process in product development. Furthermore, Eraut et aL (1975) had applied a set of 
criteria to judge the functions of descriptive analysis, evaluation and decision-making 
inherent in instruments used to analyse curriculum materials. These two elements were 
then adapted to form an instrument to measure the characteristics and the functions 
occurring in techniques applied to evaluate instructional materials. 
Development of this instrument passed through three main stages. Initially, those 
characteristics described were ordered and presented on a written form. From this form, 
a checklist v/as then developed and presented to a colleague with expertise in 
measurement and statistics. On the basis of comments, this checklist was revised into a 
rating form. This instrument is reproduced as Appendix B. 

2.2.1.2.3 Data Collection Methods 

2.2.1.2.3.1 Documents 

A survey of official documents, published by a variety of sources, was considered to be 
an important source of information for determining current practices on the uses of 
educational products in the instructional program. These documents included policy 
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statements issued by the Education Department of Tasmania and documents resulting 
from projects. 

A second class of document was instruments used to screen or to evaluate instructional 
materials. In some cases, sample evaluations produced through the use of an instrument, 
were also examined. The uses of these instruments are reported in conjunction with 
results of interviews. 

2.2.1.2.3.2 Interviews 

Two techniques were employed to collect data: firstly, a form was devised which included 
the major elements of the measurement instrument; and secondly, the same form was 
adapted to serve as an interview schedule. This form is reproduced as Appendix C. 
Initially, a copy of the form was circulated by mail to a representative of each service 
agency. The completed form was then returned by mail. Through use of the form, the 
researcher sought two types of information: firstly, the extent of the practices used by 
the staff of the service agency to evaluate instructional materials; and secondly, a request 
for subsequent information through an interview or documentary evidence. 
The interviews were conducted at mutually arranged times between the researcher and 
contacts at each of the sites. The interviews, which in all cases required from a half to 
one-and-a-half hours to complete, were conducted at each site by the researcher who 
recorded information in writing. Because interviews were arranged to suit the 
convenience of respondents, it was impracticable to use recording devices. 

2.2.1.2.4 Data Analysis 

2.2.1.2.4.1 Documents 

Qualitative analysis was applied to data obtained from examim'ng documents. Once each 
document had been read, attention was given to the emphases or omissions of particular 
key issues. On this basis, the researcher was able to form intuitive judgements on the 
conceptual understandings shown by writers of documents. 

2.2.1.2.4.2 Interviews 

Qualitative analysis was also applied to data obtained from interviews. At a time shortly 
after the conduct of each interview, the researcher prepared a record of the interview 
based upon written notes taken during the course of the interview. Each interview record 
took the form of the interview schedule. As a check on accuracy, a copy of the interview 
record was sent to each respondent, who consented to participate in this procedure. In 
each case, the interview record was then revised on the basis of the respondent's 
comments. 

It was necessary at this stage both to reduce the quantity of information to be reported 
and to administer the instrument the researcher had developed to assess each technique. 
Administration of the instrument allowed the researcher to reduce the quantity of 
information so that, in its final presentation, it was reported in tabular form. Following 
administration of the rating form, the results obtained were transcribed onto a reporting 
form suitable for reporting the results. 

2.2.1.3 The Results 

2.2.1.3.1 Documents 

In Tasmanian education, techniques for selecting and evaluating curriculum materials 
have been introduced during the last decade. This has been accomplished for print 
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materials, audio-visual materials and educational equipment through a resource 
management policy. Introduced in 1975, the Resource Management Program comprises 
three sequential steps: identification by a school's staff of problems in the management 
of resources in the school; implementation of techniques for resource management; and 
information to maintain resource management in the school 

The Curricurum Resources Section of the Education Department of Tasmania provides 
a service to schools for reviewing resource management. The service provided by its staff 
applies techniques developed from the Resource Management Program. Following an 
inquiry from a school, consultants identify variables that affect resource management in 
that school, consult teachers and implement techniques described in the Resource 
Management Guide for Australian Schools. 

2-2.1.3.1.1 The Resource Management Guide for Australian Schools 

To support the implementation of the resource management policy, the Education 
Department of Tasmania (1983) has developed a guide comprising the following 
components: a user's guide; an introductory paper. Introduction to Resource 
Management; ten photographic study cards; a chart. Investigating Resource 
Management; ten Implementation Papers; and a 16mm film, Mrs. Harding Teaches 
Resourcefully. 

The Implementation Papers are titled. Determine priorities for the acquisition of 
learning materials and equipment. Co-ordinate the acquisition of learning materials and 
equipment. Include school-produced learning materials when planning acquisitions. 
Co-ordinate decentralised facilities and collections, Intershelve audiovisual resources. 
Organise teacher-produced learning materials. List resources centrally. Match learning 
materials to the teaching program, Basic audiovisual equipment requirements for a 
school and its organisation and maintenance, and Teach information skills. Although 
these components are designed to be used collectively to conduct field work in resource 
management, the user's guide can be used independently for this purpose. 
The components of the material are designed to develop and to implement a resource 
management program that follows a particular sequence over a specified duration. Step 
1 is designed to run for two hours duration at a single session with a school's staff. Tne 
objective of Step 1, Recognition, is "to encourage staff to voice their concerns about 
resources in the school and to recognise the existence of problems" (Education 
Department of Tasmania, 1983, 11). Within a group session, a school's staff members 
read the introductory paper. Introduction to Resource Management, and then view the 
film, Mrs. Harding Teaches Resourcefully, In small groups, the participants then 
consider each of the study cards to identify the underlying issue of each and to determine 
its relevance to their own situation. The small groups then reunite to discuss and to 
compare the various issues identified. The terminal outcome is to elect a small committee 
to implement Step 2. 

The objectives of Step 2, Investigation, are "to clarify the strengths and weaknesses in the 
acquisition, organisation and use of resources in the school" and "to determine the areas 
of resource management which the school wishes to improve" (Education Department 
of Tasmania, 1983, 27). Members of the committee, which should include either the 
school's principal or vice-principal, meet to conduct several activities on resource 
acquisition, resource organisation and resource use. These activities are to research and 
to document information on existing resource management in the school; to discuss and 
to interpret this information; and to decide which aspects of resource management need 
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improvement. A chart, Investigating Resource Management, is provided to complete 
this task. 

The objective of Step 3, Implementation, is "to provide guidelines for implementing 
improvements in selected aspects of Resource Management" (Education Department of 
Tasmania, 1983, 33). Step 3 is achieved through the principal appointing an 
implementation committee, which has the task of selecting, in an order of priority, 
possible improvements for resource management. Sequential steps to be taken by the 
committee are to consider suggested procedures for implementation by referring to the 
appropriate Implementation Paper, and to decide on a suitable implementation strategy 
for the .^elected project. 

2.2.1.3.1.2 Library Collection Development in Schools and Colleges 

The Education Department of Tasmania (1985) has also published a policy statement. 
Library Collection Development in Schools and Colleges. The developer of the policy 
statement proposed that the staff of each Tasmanian state school should develop a policy 
which includes statements on selection criteria and selection procedures. It is also 
suggested each school should establish a Curriculum Resources Development 
Committee, which should meet three times a year, to evaluate curricular needs for 
materials, to evaluate existing curriculum materials, to select and to acquire new 
instaictional materials, and to maintain collections. 

2.2.1.3.1.3 The Learning Resources Review 

During 1985 and 1986, a review of the uses of instructional materials in Tasmanian schools 
was conducted by the Education Department of Tasmania. Because the provision of 
instructional materials represents an expensive investment, the main purpose of this 
review was to determine accountability. An additional purpose of the review was to 
evaluate the impact of the work of the Resource Management Team, established in 1978 
to assist school staffs implement effective systems for managing their collections of 
educational products. The review had four main aims: to review the existing procedures 
used by school staffs to select instructional materials; to review the existing systems used 
in schools to organise their collections of instructional materials; to review the 
fnechanisms whereby curriculum materials provided by the Education Department of 
Tasmania are made available to'schools; and to prepare recommendations on how both 
the delivery of instructional materials to schools and their management in schools can be 
improved. 

At the time of writing this report, the review is still in progress although a final report. 
Education Department of Tasmania (n.d.) has been prepared. The review commenced 
with the appointment of a committee of thirteen members drawn from all sections of the 
Education Department of Tasmania, which first met during August 1985. From twelve 
officers of the Curriculum Resources Section and four advisory staff, an executive group 
of six members was formed to facilitate an evaluation project. 

The executive group chose to employ a multi-site case study design to collect data. The 
case studies involved seventeen schools and twenty-five resource centres. Fourteen 
people, working in teams of two, collected data through the use of interviews and 
questionnaires during a seven-week period in April and May, 1986. Felton and Johnson 
(1986) have reported upon the application of a set of techniques in this project to increase 
the reliability of data collection, data analysis and the reporting of data through reduction 
and display. 
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The preliminary report was prepared during September and October 1986 on the basis 
of the reporting of the data at a workshop. In this report, certain criteria - 
appropriateness, leadership, coherence, centralisation, equity and support - were 
identified for evaluating the effectiveness of resource management. It was then 
determined that the ciiterion, appropriateness, comprised several characteristics: 
funding; selection; acquisition; storage; cataloguing; circulation; communication; 
production; sharing; planning; evaluation; and staffing. 

On the basis of the interpretation of the results, the committee presented twelve 
recommendations to the Education Department of Tasmania and to colleges and to 
schools. Each of those recommendations that relates to the uses of educational products 
is listed under the appropriate section. 
Power and Leadership 
Recommendation 2 states: 

That a professional development program for school and college principals and senior staff 
be planned and implemented, incorporating at least the following components: 

a. matching curriculum needs with the selection, acquisition, organisation and use of learn- 
ing resources; 

b. developing and implementing a written learning resources policy statement; 

c. the essential elements of efficient resource management". 
Recommendation 5 states: 

"That the Education Department enter into discussions with the Centre for Education of the 
University and with the Tasmanian State Institute of Technology with the view to increasing 
and improving components on learning resources and their management in Initial teacher 
education". 

Centralisation 

Recommendation 7 states: 

'That schools and colleges act with all possible speed to create a central listing of all cur- 
riculum resources". 

(Education Department of Tasmania, n.d. Learning Resources Review, Hobart; Education 
Department of Tasmania, 28) 

2.2.1.3.2 Interviews 

Two problems were tested in this multi-site case study: firstly, that there would be an 
extensive application of informal procedures by the staffs of service agencies which, in 
consequence, would reduce reliability of measures; and secondly, that there would be 
incomplete application of the set of techniques in the sequence which, in consequence, 
would reduce the validity of measures. 

A summary checklist of techniques performed on site-developed instructional materials 
at each service agency is shown in Table 1. It is evident from an examination of this table 
that techniques are applied to site-developed instructional materials at nine service 
agencies (52.9%). In these cases, it is apparent that the coverage of techniques at each 
step in the sequence is partial. Whereas procedures for both learner-based verification 
and revision, and decision-making for implementation were performed widely, few 
procedures were used for screening, for descriptive analysis and evaluation, and for 
decision-making for selection of site-developed instructional materials by the staffs of 
these service agencies. 

The descriptive characteristics of each technique applied at the nine service agencies are 
shown in Table 2. The descriptions indicate that at least one technique, and sometimes 
two or more techniques, for providing information on learner-based verification and 
revision is performed at eight service agencies (88.9%). The usefulness of these 
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TABLE 1 



SUMMARY CHECKLIST OF TECHNIQUES PERFORMED ON SITE-DEVELOPED 
INSTRUCTIONAL MATERIALS AT SERVICE AGENCIES. 



Criterion Site I Site2 Site3 Site4 Site5 Site6 Site? SiteS Site9 SitelO Sitell Sitel2 Sitel3 Sitel4 Sitel5 Sitel6 Sitel? 
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Note 1. na = not applicable 
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TABLE 2 



TECHNIQUES APPLIED TO SITE-DEVELOPED INSTRUCTIONAL MATERIALS AT SERVICE 

AGENCIES. 



Criterion 


Sitel Site2 SiteS 


Site4 Site5 Site6 Site? SiteS Site9 SitelO Sitell Sitel2 SitelS Sitel4 Sitel5 Sitel6 Sitel? 


Learner- 
based 
verification 
and revision 


PR^ 
PE^ 
FR^ 
FE-* 


FB} Pr2 fr^ pr3 pr3 Pr2 

fr3 fr- 


Screening 
for adoption 


Descriptive 
analysis and 
evaluation 




DE^ 


Decision- 
making 
for selection 




IP IP IP 


Decision- 
making 
for 

imnlft- 
mentation 


IP 
DM 


IP IP DM IP IP IP IP 

DM DM 



Note 1. Key to techniques ' 

IP = informal procedure DE = descriptive analytical and evaluative review 

ID = instructional design analysis 

IC = in >tructional design analysis to establish internal congruence 
EC = instructional design analysis to establish external congruence 
AN = annotation 
WB = written brief 
RS = rating scale 
DM = demonstration 

Note 2. The extent to which the technique meets a minimum standard of validity on a Likert-type rating scale. 



PR = 


pilot-trial 


PE = 


pilot-test 


FE = 


field-test 


FR = 


field- trial 


AF = 


appraisal form 


Da = 


descriptive analytical review 



1 = yes, definitely valid 
3 = of uncertain validity 
5 = no, definitely invalid 



2 = yes, probably valid 
4 = no, probably invalid 
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TABLES 



SUMMARY CHECKLIST OF TECHNIQUES PERFORMED ON COMMERCIALLY PRODUCED 
INSTRUCTIONAL MATERIALS AT SERVICE AGENCIES. 



Criterion Sitcl 


Site2 


Sitc3 
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Site? 
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SitelO Sitell Sitel2 Sitel3 Sitel4 SitelS Sitel6 Sitcl? 
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TABLE 4 



TECHNIQUES APPLIED TO COMMERCIALLY PRODUCED INSTRUCTIONAL MATERIALS 

AT SERVICE AGENCIES. 



Criterion Sitel SiteZ Sitc3 Site4 SiteS Site6 Site? SiteS Site9 SitelO Sitell Sitel2 SitelS SiteM Sitel5Sitel6Sitel7 
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RS 



IP IP 



IP 
RS 



IP IP IP 



IP 



Decision- 
making 
for 

imple- 
mentation 



IP WB IP 
DM 



IP 



Note 1. Key to techniques 

IP = informal procedure 

PR = pilot-trial 

PE = pilot-test 

FE = field-test 

FR = Hald-trial 

AF = appraisal form 

DA = descriptive analytical review 



IP WB IP 
DM DM DM 



IP DM IP IP IP IP 



DE = descriptive analytical and evaluative review 
ID = instructional design analysis 

IC = instructional design analysis to establish internal congruence 
EC = instructional design analysis to establish external congruence 
AN = annotation 
WB = written brief 
RS = rating scale 



Note 2. The extent to which the technique meets a minimum standard of validity on a Likert-type rating scale. 



1 = yes, definitely valid 
3 = of uncertain validity 
5 = no, definitely invalid 



2 = yes, probably valid 
4 = no, probably invalid 
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SUMMARY CHECKLIST OF TECHNIQUES PERFORMED ON COMiviERCIALLY PRODUCED 

EDUCATiONAL EQUIPMENT 
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Note 1. na = not applicable 



TABLE 6 



TECHNIQUES APPUED TO COMMERCIALLY PRODUCED EDUCATIONAL EQUIPMENT 

AT SERVICE AGENCIES 


Criterion 


Sitel Site2 Site3 Site4 SiteS 


Site6 Site? SitcS Site9 SitelOSitellSitel2Sitel3Sitel4Sitel5Sitel6 Sitel? 


Learner- 
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Note 1. Key to techniques 

IP = informal procedure 
PR = pilot-trial 
PE = pilot-test 
FE = field-test 
FR = field-trial 



AF = appraisal form 
DA = descriptive analytical review 
NAV = performed but not available for publication 
Note 2. The extent to which the technique meets a minimum standard of validity on a Likert-type rating scale 



DE = descriptive analytical and evaluative review 
ID = instructional design analysis 

IC = instructional design analysis to establish internal congruence 
EC = instructional design analysis to establish external congruence 
AN = annotation 
WB = written brief 
RS = rating scale 



1 = yes, definitely valid 
3 = of uncertain validity 
5 = no, definitely invalid 



2 =: yes, probably valid 
4 = no, probably invalid 
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techniques ranged from probably valid to probably invalid. None of these service 
agencies applied techniques to screen site-developed instructional materials, and only 
one service agency (11.1%) applied a technique for their descriptive analysis and 
evaluation. Three service agencies (33.3%) applied techniques to enable teachers to 
select site-developed instructional materials whilst eight service agencies (88.9%) 
applied techniques to assist teachers to implement such materials. In ten (71.4%) of 
fourteen cases, techniques characterised as informal procedures, based upon a 
consultative process between a consultant and a teacher, were used for decision-making, 
although classroom demonstratioi. was used for implementation in four instances 
(28.6%). 

A summar, checklist of techniques performed on commercially produced instructional 
materials at each service agency is shown in Table 3. It is evident from an examination 
of this table that techniques are applied to commerciallyproduced instructional materials 
at sixteen service agencies (94.1%). It is apparent that the coverage of techniques at each 
step in the sequence is extensive. Although few servace agencies applied techniques of 
learner-based verification and revision to commercially produced instructional materials, 
the subsequent steps of screening, descriptive analysis and evaluation, and 
decision-making for both selection and implementation, were well represented. 
The descriptive characteristics of each technique applied at the sixteen service agencies 
are shown in Table 4. Tlie descriptions indicate that only two service agencies (11,8%) 
provide information to publishers on learner-based verification for revising commercially 
produced instructional materials. Twelve service agencies (75.0%) apply techniques to 
screen commercially produced instructional materials, although, in all but'one case, these 
techniques are informal procedures based upon inspections of materials or.,oiisultations 
between consultants. Ten service agencies (62.5%) apply techniques for descriptive 
analysis and evaluation of commercially produced instructional materials, which 
generally appear to convey valid information on specified characteristics. Of the fifteen 
service agencies, which apply techniques to assist teachers to select commercially 
produced instructional materials, fourteen service agencies (93.3%) use informal 
procedures based upon consultative processes between a consultant and a teacher, whilst 
soinetimes this is supplemented by use of a rating scale. Of the thirteen service agencies, 
whicii apply techniques to assist teachers to implement commercially produced 

consultative processes between a consultant and a teacher, whilst classroom 
demonstrations are used by five service agencies (38.5%) and written briefs are used by 
two service agencies (15.4%). In thirty-five (74.5%) of forty-seven cases, informal 
■procedures were used for screening and decisi -^-making. 

A summary checklist of techniques performed on commercially produced educational 
equipment at each sen/ice agency is shown in Table 5. It is evident from an examination 
of this table that techniques are applied to commercially produced educational 
equipment at only two service agencies (11.8%). In the single case available, the coverage 
of techniques at each step in the sequence is partial. 

The descriptive characteristics of each technique applied at this single service agency are 
shown in Table 6. The descriptions indicate that informal procedures are used in all 
instances for screening and decision-making. 

2.2.1.4 The Discussion 



2.2.1.4.1 Documents 

The evidence, provided by an examination of the documents pubHshed by the Education 
Department of Tasmania on issues related to the use of curriculum materials, shows that 
overwhelming attention is given to resource management. Resource management is 
discussed in terms of both presenting a coherent policy and a set of practices for schools 
and resource centres. 

Incorporated within the resource management policy, however, is a set of practices on 
the selection and the evaluation of instructional materials. On various occasions, the 
treatment of the selection and the evaluation of instructiongl materials is couched in 
terms that relate to either resource management of collections or to their use in the 
curriculum. 

On adopting, evaluating and selecting instructional materials, the authors state in 
Implementation Paper No. 2, Co-ordinate the acquisition of learning materials and 
equipment, that: 

"It is important that standardised procedures are adopted for the selection of resources... A 
workable system may operate through the following stages: 

(i) Requests for the purchase of a particular item should be submitted to the Resource Co- 
ordmator who checks that the item is not an unnecessary duplication of an existing resource 
and that a suitable alternative is not readily available. A central listing of all resources avail- 
able to the school will facilitate this step. 

(ii) Every effort should be made to obtain a valid evaluation of the resource requested. Such 
evaluation ideally is the first-haiid previewing of the item by the prospective user, or where this 
is not practicable, an objective and reputable review of the item. 

(iii) The request, once verified, passes to the Resomc? Acquisition Committee which should 
ensure that the resource matches the resource needs already established and then allocate 
funds or a funding priority". 

(Education Department of Tasmania, 1983 Resource Management Guide for Australian 
Schools, Hobart: Education Department of Tasmania, 36) 
The process for matching instructional materials to the curriculum has been identified 
but is inadequately related to the sequence of steps leading from the adoption, the 
descriptive analysis and evaluation, the selection, to the implementation of materials. In 
Implementation Paper No. 8, Matching learning resources with the curriculum, three 
elements are identified in this process: the objectives match; the context match; and the 
media match. As described, an objectives match is intended to match an instructional 
material to a teacher's instructional objectives, talcing account of variables such as 
readability and learnability. In a context match, techniques are applied that are likely to 
identify instructional materials most appropriate to a particular educational context. The 
intention of a media match is to improve the use of educational products in a variety of 
media through the use of appraisal forms. 

Although the authors of this document have identified two fundamental constituents in 
a process to match curriculum materials to instructional programs, an effective 
connection between these constituents has not been established. Furthermore, the 
techniques that teachers and media personnel are to apply to realise this are unlikely to 
reach minimum standards of reliability. There is no application of instructional design 
analysis to establish either internal or external congruence, which need to be determined 
to achieve intended matches between instructional materials and instructional programs. 
This general conclusion is confirmed by statements made in the Learning Resources 
Review. In Recommendation 2, the authors of this document recognise the need to 
improve principals' understandings of the relationship between curriculum needs and 
adoption, selection and evaluation procedures through inservice teacher education. This 
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relationship, however, is only established in terms of the provision of appropriate 
resource management programs in schools and not to applying a technique of 
instructional design analysis to match curriculum materials and instructional programs 
validly and reliably. Despite this conclusion, the authors state that: 

"Although the Review, then, concentrates its attention on the mechanics of resource 
provision, organisation, management and distribution, its importance lies in the effect it may 
have on the improved use of curriculum resources in learning and teaching. The ways in which 
students and teachers actually do use such resources would be the subject of a valuable and 
complementary review". 

(Education Department of Tasmania, n.d. Learning Resources Review, Hobart: Education 
Department of Tasmania, 8) 

2.2.1.4.2 Interviews 

The problems tested in this multi-site case study were that, with an extensive application 
of informal procedures, there would be lower reliability from measures than is necessary 
to reduce random errors to an acceptable level, and that, with an incomplete application 
of the set of techniques in the sequence, there would be lower validity from measures 
than is necessary to reduce systematic errors to an acceptable level. Neither of these 
problems could be tested by experimental design because of the qualitative nature of the 
data. The relationships, firstly, between the use of informal procedures and low reliability 
and, secondly, between the imperfect sequence of techniques and low validity, are based 
upon assumptions. Therefore, this discussion, supported by findings related to the 
problems investigated, is based upon judgements made on the part of the researcher. 
The results of the study showed that, on the one hand, there were extensive applications 
of informal procedures and criteria which did not appear to relate to the instructional 
design whilst, on the other hand, techniques were not applied consistently at each step of 
the sequence. These results are consistent with the only research hitherto published on 
practices of selection and use of curriculum materials in Australian schools. This research 
was originally reported by Marsh et al. (1981). These researchers then reported on the 
attitudes of Western Australian primary school principals towards the selection of 
curriculum materials (Marsh, 1983a), on attitudes of social studies instructors, surveyed 
nationally at institutions of further education, towards including curriculum materials 
analysis in pr eservice teacher training courses (Marsh, 1983b), and on Western Australian 
teachers' attitudes towards selection issues involving curriculum materials (Marsh et al 
1985). 

One can only speculate on the reasons why the staff members of service agencies either 
rely upon informal, intuitive procedures, rather than use more reliable techniques, or fail 
to apply techniques more consistently so that each step in the sequence is taken into 
account. In the former case, it can be surmised that most subjects surveyed in the case 
study adapt informal procedures to suit their contextual requirements, but do not seem 
to be aware of the availability of more reliable techniques. In the latter case, most subjects 
did not appear to recognise that an inherent sequence of steps for analysing instructional 
materials prevails between the stages of product development and implementation. One 
can conjecture that such informal procedures have developed spontaneously as practical 
mechanisms, based upon inherently generic principles. 

A limitation, imposed upon studying the variables associated with the analysis of 
instructional materials and the application of this information to their uses in classrooms, 
is the qualitative characteristics of much of the data. State-of-the-art methods for 
researching these attributes have not identified variables capable of treatment through 




statistical analysis, but rather they have examined invariable elements that are more 
appropriately analysed qualitatively. Quantitative analysis is currently restricted to 
measuring the amounts of these elements. Consequently, the design of this multi-site 
case study was confined by this restriction and, therefore, it is impossible to provide the 
kind of quantitative data which would make the use of experimental design a valuable 
tool for investigating manipulatory variables. 

2.2.2 Evidence from Other Australian States 

2.2.2.1The Researcii Problem 

The researcher also decided to elicit information on techniques to evaluate educational 
products used by education departments in other Australian states and territories. It was 
believed that descriptive assessments of these techniques would enable the researcher to 
extrapolate, to a national setting, the findings of documentary evidence and the multi-site 
case study undertaken in the Tasmanian context. 

2.2.2.2 The Method 

To obtain this information, appropriate staff members were identified in other educafion 
departments: the New South Wales Department of Educafion; the Ministry of 
Education, Victoria; the Education Department of South Australia; the Queensland 
Department of Education; the Ministry of Educafion, Western Australia; the Australian 
Capital Territory Schools Authority; and the Northern Territory Department of 
Education. These staff members were then contacted through correspondence. 

2.2.2.3 The Results 

Informafion on procedures to evaluate educafional products was obtained from the New 
South Wales Department of Education, the Ministry of Education, Victoria, the 
Queensland Department of Educafion, the Educafion Department of South Australia 
and the Ministry of Educafion, Western Australia. The results of this survey are now 
reported in detail. 

2.2.2.3.1 New South Wales Department of Education 

A report by the New South Wales Department of Education (1981) led to the 
establishment of two committees, the Equipment Committee and the Learning Materials 
Committee, which recommend adoptions of educational products to a Supply 
Co-ordinating Committee. Each committee undertakes a similar procedure for adoption 
of educational products: idenfification; screening; advising on needs; inifiating 
resear ch; advising on disseminafion of information on products; and recommending the 
establishment of working parties. Members of these committees co-operate with 
Curriculum Project Teams and Curriculum Implementation Co-ordinating Groups in 
order to gain access to informafion on commerical products. The Curriculum Project 
Teams and Curriculum Implementation Co-ordinating Groups also recommend 
development of non-commercial, instrucfional materials to the Director of Services, New 
South Wales Department of Education. 

Once an educafional product has been adopted, a descriptive analyfic and evaluative 
review, termed a specificafion, is undertaken by a staff member of the New South Wales 
Department of Education. This specificafion is based upon the following criteria: 
educational rationale - for what purpose the item is needed in schools, how the item is to 
be used in schools, where in schools the item is to be used, and what personnel in schools 
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will use the item; performance objectives; boundary conditions - a statement of physical 
description which is judged against a specified minimum or maximum standard; and 
technical details. 

2.2.2.3.2 The Ministry of Education, Victoria 

The Ministry of Education, Victoria (1984) has published a guide to facilitate the 
appraisal of audio-visual materials. A set of guidelines, consisting of three parts, is 
provided for descriptive analysis and evaluation of audio-visual materials: educational 
considerations (educational objectives, total effect, curriculum area, appropriate level, 
style, continuity, running time, appropriate medium, economic justification, and 
utilisation); technical considerations (image quality, techniques, colour, sound); and 
other factors (aesthetic considerations of the context). 

2.2.2.3.3. The Queensland Department of Education 

As part of the Library and Resource Services Branch of the Division of Curriculum 
Services, the Queensland Department of Education has established a Resource 
Evaluation Team. This team uses an appraisal form to screen basic instructional 
materials, supplementary instructional materials, audio-visual materials and computer 
courseware. Separate appraisal forms are used by consultants to screen curriculum 
materials for special needs groups. Two forms, one for print materials and one for 
non-print materials, are used for descriptive analytic and evaluative reviewing of 
instructional materials. Reviews are provided for all subject areas across all grades. A 
journal, Reviewpoint, is published to report these reviews to teachers in both state and 
independent schools to aid decision-making by selectors. 

The form. Resource Evaluation - Books, consists of five sections: bibliographic 
information; content; presentation; suitability (grade level, special needs groups, 
curriculum role); special features (physical description); and reviewer's response 
(recommendation for selection). The form. Resource Evaluation - Non-book Material, 
consists of six sections: description of item (components); method of evaluation; subject 
matter (contents); level; ratings (quantitative ratings on teachers' notes, student guide, 
photography, sound organisation and overall rating); and general evaluation (on 
accuracy, currency, bias, general appeal, relevance to courses and technical aspects). 
A standard appraisal form is used to screen educational equipment for purchase. 
Quantitative ratings on sixteen criteria relating to technical conditions of the equipment 
are incorporated in this form. Ratings are made on the basis of information completed 
on separate checklists. The checklists are used also to compile descriptive analytic and 
evaluative reviews of audio and video hardware and computer hardware, which are 
published in a Resources Catalogue. 

The Schools Computing Service uses a form. Computer Software Report, to screen and 
to compile descriptive analytic and evaluative reviews of courseware, which are presented 
to teachers through a journal. Sugar. This form consists of five sections: curriculum 
match; program description (instructional mode, general description, educational 
objectives); presentafion (rating scale on technical aspects); overall comment 
(recommendation for adoption); and special notes. 

2.2.2.3.4 The Education Department of South Australia 

The Education Department of South Australia (1987) has published a set of guidelines 
for the selecfion and the access of educational products. Selection of educational 
products should be guided by eight principles: to relate materials to a school's curriculum 
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policy and instructional program; to support a curriculum which fulfils pluralistic aims; 
to encourage multiculturalism; to motivate positive attributes towards citizenship; to 
be relevant to students' needs; to provide for discrimination in students' choices; to 
represent a range of views; and to be based upon the relative strengths of materials. 
Specific guidelines, which follow a sequential procedure, are recommended to schools' 
staffs: development of a selection policy; responsibilities of teachers, students and 
parents; and developing guidelines for implementation, which includes frameworks for 
selection and review. 

The Educational Production Service of the Education Department of South Australia 
uses Evaluation Form - 16mm FilmA^ideo to appraise audio-visual materials for 
adoption. 

2.2.2.3.5 Ministry of Education, Western Australia 

School personnel and consultants use a set of three, parallel forms - Primary Fiction 
Appraisal Form, Non-fiction Appraisal Form K-12, and Non-book Appraisal Form K-12 
- to provide descriptive analytical and evaluative reviews, reported to teachers to aid 
decision-making for selections. A journal. Fiction FocuSy is published to report reviews 
of supplementary instructional materials to teachers. 

Each form in the set consists of four sections: criteria for identifying the medium of the 
material; criteria, based upon rating scales, for evaluating production quality; criteria, 
based upon rating scales, for evaluating content appraisal (product description); and 
criteria for descriptive analysis of the contents and biases in the material. Annotated 
background information is added to each descriptive analysis. Each of the forms is applied 
to develop descriptive analytical and evaluative reviews. 

2.2.2.4 The Discussion 

Although the small number of documents and instruments provided by Australian 
education departments, outside Tasmania, does not comprise a representative sample, 
there is sufficient variety in these materials to allow the researcher to form judgements 
on the characreristics of techniques used to select and to evaluate educational products. 
An examination of these documents and instruments supports a judgement that the 
characteristics of the practices recommended for the selection and the evaluation of 
educational products are similar to those undertaken in the Tasmanian context. There 
is, however, greater attention given to the provision of selection policies that will meet 
the needs of the curriculum and instructional programs. On the other hand, the emphasis 
upon resource management is absent, although its implications are taken into account. 
The instruments used for descriptive analysis and evaluation of educational products 
reflect procedures that are similar to those used in Tasmanian education. The highest 
level of application achieved in any one of these instruments was descriptive analytical 
and evaluative review. 

2.3 Conclusion 

The evidence provided by this survey shows that reliable techniques for adopting, for 
evaluating, for selecting and for utilising educational products have yet to be applied 
widely in Australian education. This conclusion is supported by the data gained on 
procedures for selection and use of instructional materials in Australian schools provided 
by Marsh et al. (1985). 
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In part, the responsibility for the prevailing inadequacy must be placed with the failure 
of Australian educational agencies both to address the evaluative issues satisfactorily and 
to co-ordinate their activities effectively. In the past, the failure of either The Australian 
Council for Educational Research or the Curriculum Development Centre to accomplish 
an outcome useful to Australian education appears to be as much a consequence of an 
unfortunate combination of circumstances as ineffectual actions. Already alluded to 
earlier in this chapter, this combination of circumstances has resulted in neither 
organisation assuming responsibility for initiating an effective research and development 
program resulting in the implementation of an innovative service to Australian schools. 
In spite of the work on instructional design analysis undertaken by Jeffery, and Piper 
(1976) at The Australian Council for Educational Research and the attention given to 
this issue by the Curriculum Development Centre in publications (Curriculum 
Development Centre, 1977; and Davis, 1981), it seems that neither organisation has been 
capable of capitalising upon such activities to translate this research and development 
into provision of a service to Australian schools. It is apparent that the substantial costs 
of investment, firstly, in research and development and, secondly, in disseminating the 
innovation, have been perceived as beyond the funding provisions of Australian 
education in the past. The establishment of a national database of annotated information 
on instructional materials through the Australian Schools Catalogue Information Service, 
however, now makes the development of a nationwide service of evaluations of 
educational products an attainable proposition both financially and technically. 
The aspects of the survey that relate to the national context have indicated that Australian 
education, however, is generally lacking a foundation of research and development 
activities upon which to support implementation of such a service. Tnis verdict is verified 
by empirical evidence gathered from state education departments in Australia. The 
instruments and procedures used are characterised by their developers' failures to take 
account of current understanding on the sequence of techniques, and the criteria used 
do not appear to relate to instructional practices. Two trends, identifiable at the state 
level, indicate that it is imperative in the immediate future to improve the reliability and 
validity of such techniques applied to instructional materials. Firstly, state education 
departments in several of the more populous Australian states are implementing 
organisational frameworks to provide statewide evaluations of educational products; and 
secondly, there is an emerging awareness of a need to rationalise such activities. 
Empirical findings derived from the survey of service agencies of the Education 
Department of Tasmania identified the widespread use of informal procedures which, 
however functional and useful in specific educational contexts, are not supported by the 
comprehensive application of techniques that would preside standardised and reliable 
information on matching instructional materials to the instructional design of educational 
programs. Generally, techniques for learner-based verification and revision are only 
extensively applied to curriculum materials developed by the Education Department of 
Tasmania; there is no evidence to support either the consistent use of standardised 
criteria or the necessity to report the results of pilot-tests or field-tests. Screening of 
commercially produced materials is almost entirely dependent upon the use of informal 
procedures with the predominant criterion based upon the material's intrinsic philosophy 
matching the prevailing educational philosophy. Although several examples of 
competently produced descriptive analytic and evaluative reviews were identified during 
the survey, the respondents appeared to be ignorant that more reliable techniques, based 
upon instructional design analysis, are available. Both the steps of selection and 
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implementation depend upon the use of informal procedures. Rating scales, usually 
appended to the descriptive analysis and evaluative review of a material, are sometimes 
used to present recommendations for selection. Classroom demonstrations are often 
used to recommend the implementation of materials. These observations suggest that 
the staff members of service agencies are failing to report both reliable and valid 
information on instructional materials to teachers, probably for two reasons: the 
techniques they apply do not relate such information to the instructional design; and an 
emphasis is placed upon managing curriculum resources rather than tackling the more 
pertinent issues of accomplishing a better match between curriculum materials and 
educational programs in order to improve student achievement. 
Marsh et al. (1985) have provided evidence on the practices of selection of curriculum 
materials prevailing at primary schools in Western Australia, which gives credence to the 
contention that the inadequacies which occur at service agencies in Tasmania are 
widespread. Three designs were administered in two stages to collect data in this project. 
The first stage comprised two designs: firstly, case studies were conducted in two schools; 
and secondly, structured interviews were conducted with teachers and principals in 
selected schools. The data collected at the first stage were used to develop two 
questionnaires for the second stage. These questionnaires, one each for gathering 
information on social studies and mathematics, were administered to separately drawn, 
simple random samples of forty schools. Each questionnaire was designed in alternate 
forms, one for principals and the other for teachers. 

The results from the. surveys of teachers identified problems reflecting inadequacies of 
the prevailing selection practices in the schools studied for both social studies and 
mathematics. These problems included lack of time to select curriculum materials, not 
knowing which curriculum materials were available and lack of access to curriculum 
materials. Principals also experienced problems in selecting curriculum materials, which 
included difficulties in selecting materials to suit most teachers in their schools, the 
availability of too limited amounts of funding for purchases, and being required to order 
materials without preliminary inspections. Furthermore, the results suggested that status 
position, as much as role performance, influenced the selection practices in these 
schools. Although only a fifth of principals had sole responsibility for orgardsing 
instructional materials in their schools, principals often formed part of a staff group that 
performed this task, particularly in relation to mathematics materials. On the other hand, 
both teacher-librarians and library aides took a major role in assisting teachers to select 
instructional materials for social studies. It was also identified that approximately 
one-third of teachers surveyed were involved in making decisions on selecting 
instructional materials for their classrooms. 

These researchers concluded that there appeared to be insufficient consultation between 
principals and their staffs on the selection of instructional materials, which may have 
resulted, firstly, from a lack of school policies concerning the selection of materials, and 
secondly, from teachers having failed to identify materials which they considered to be 
the most appropriate for use in their classrooms. This situation is understandable 
considering that teachers would be only able to rely upon intuitive process to provide this 
sort of information. 

The evidence from the survey has shown that the staff members of service agtj^c.'os of 
state education departments responsible for providing this information to teachers do 
not apply techniques that are capable of achieving a valid match between instructional 
materials and instructional programs. One can predict that a substantial improvement 
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to the situations in both service agencies of state education departments and schools in 
Australia can be best initiated through research and development on adapting techniques 
available from foreign sources to Australian conditions, through field-testing and 
modifying procedures to establish a national database of information on educational 
products, and through disseminating this information to all sections of the educational 
community in Australia. 
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CHAPTERS 



THE AMERICAN CONTEXT 



In this chapter, the author provides a descriptive account of four American educational 
organisations, which have as their missions, evaluating and selecting instructional 
materials and researching issues involved in the uses of educational products. The 
organisations are the Educational Products Information Exchange (EPIE) Institute, the 
Social Science Education Consortium (SSEC), the Council on Interracial Books for 
Children (CIBC), and the Women on Words and Images (WOWI). 
This account is then followed by a comprehensive survey of literature on these issues in 
the Resources in i'ducation (RIE) database of the Educational Resources Information 
Center (ERIC). The aim of this account is to identify educational organisations 
contributing solutions to these problems at national, state and school district levels. 
Interwoven with this survey is an account of current textbook adop^i-n practices. 
The purpose of this comprehensive survey of the American setting is to provide a 
comparative study, which may serve as a benchmark for judgements to be made on 
directions for planned change in the Australian context. 

3.1 Agencies and Programs 

The author has selected fou" American educational organisations for detailed 
examination in this section. The touchstone for selecting these four organisations was 
that the programs of each entail collecting information on print materials^ analysing print 
materials and disseminating information on print materials. 

The Educational Products Information Exchange (EPIE) Institute, the first to be 
considered, is an independent, self-supporting agency providing services exclusively 
related to these issues to paying customers on a national scale. The second, the Social 
Science Education Consortium (SSEC), also an independent, self-supporting agency, 
provides a range of programs and services on social studies and social science issues, 
including information on instructional materials. The Couuil on Interracial Books for 
Child ren (CIBC) is an independent, self-supporting agency which, in promoting 
multiracial and multicultural approaches in children's literature, provides information 
on both racist and sexist biases. The last organisation to be considered is the Words on 
Women and Images (WOWI), also an independent, self-supporting agency, which has 
conducted research on identifying sexist biases in instructional materials. 
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3.1.1 The Educational Products Information Exchange (EPIE) Institute 

The Educational Products Information Exchange (EPIE) Institute was established as an 
independent organisation on 1 August 1967, although formerly operating as a division of 
the Institute of Educational Development. Komoski (1967) described the planning for 
the EPIE Institute during its formative years in a report contracted by the Office of 
Education, U.S. Department of Health, Education and Welfare. 

3.1.1.1 The Developmental Phase (1967-1970) 

Komoski described the initial development of the EPIE Institute, as follows. At a 
meeting convened early in December 1966 by the designers of the EPIE Institute and the 
representatives of educational administrative organisations, an EPIE Advisory Board was 
constituted. Comprising thirty associations, the EPIE Advisory Board was responsible 
for auditing procedures as well as contributing to the development of the EPIE Institute. 
At the same time, some thirty consultants from a variety of sources met to advise on the 
design vof the systems to be used by the EPIE Institute, on the training of information 
gatherers, on the development of questionnaires and on methods for analysing 
instructional materials. 

During the early part of 1967, three significant issues for developing the EPIE Institute 
were considered at meetings: the definition of useful product information types and 
services; the design of a data system for the exchange of information; and the 
development of information gathering techniques. 

It was decided that three types of information should be collected by the EPIE Institute: 
product producer information; product analyst information; and product user 
information. It was determined that four modes of subscriber service should be provided 
by the EPIE Institute: a broadcast mode for transmitting generalised product 
information; a responsive mode for acting upon standard inquiries with data in the 
system; an interactive mode for dialogue between the EPIE Institute and a subscriber 
beyond standard inquiry protocols; and a customised mode for tailoring system and field 
studies. It was recognised that each of the modes could not be offered simultaneously at 
the cnnimencement of operations, so it was determined that only the broadcast mode be 
introauced inmiediately. 

The design of data system for the exchange of information was considered during the 
course of the meeting in December 1966. The design of this system for the EPIE Institute 
was contracted to the Coinmuiiity Systems Foundation, Ann Arbor, Michigan. In April 
1967, a report submitted by the Community Systems Foun on contained a proposed 
system for computer-based storage and retrieval of information. This proposal was 
modified, however, to more closely meet the operational requirements of the three-year, 
developmental phase for the EPIE Institute. 

Three aspects for developing information gathering techniques were considered by the 
consultants: assembling producer information; developing analyst information; and 
collecting user information. A process, developed for assembling information from 
producers, consisted of drawing up an exhaustive list of characteristics about a particular 
type of educational product and then submitting the list to user groups and producer 
groups for judgements. The results were discussed at a meeting at which a list of the most 
significant characteristics was drawn up. Representative school personnel were then 
consulted with the view to refining the list. 
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At the Second Lake Mohonk Work Conference, 8-10 April, 1967, the consultants 
considered the variables involved in the analysis of educational products: the use of 
checklists and prose statements; the units of analysis; the analysis of textbooks and 
supplementary materials; the dependability of inferences made by analysts; the training 
of analysts; and the development of forms. Attention at this conference was given to 
experimental work being conducted at the University of Florida and the University of 
Illinois to develop practical analytic techniques. The development of techniques to report 
on user satisfactions of teachers, administrators and students was discussed. Such 
information gathering was to be conducted by school personnel, who had received 
extensive training during August 1967. 

3.1.1.1.1 Pilot Studies 

The consideration of these issues led to the conduct of two pilot studies which were 
intended, in the first instance, to pilot-test the types and modes of EPIE services and, in 
the second instance, to field-test the EPIE system. The two pilot studies ran concurrently 
between 1968 and 1970. 

3.1.1.1.1.1 The Four-State Co-operative Project 

Once the types and modes of services were delineated, it was decided to pilot-test EPIE's 
collection procedures, its analysis techniques, its operational definitions and its 
dissemination system: The first pilot study, the Four-State Co-operative Project, was 
conducted in New York State, Pennsylvania, New Jersey and Delaware during 1968 in 
co-operation with two regional educational laboratories: the Eastern Regional Institute 
for Education (ERIE) and the Research for Better Schools (RBS). 
Instruments and techniques to examine science materials and their uses were 
administered to staffs at a sample of sixty schools selected from a population including 
all public elementary schools in these states. The project constituted the first year of 
operation of the EPIE Institute. 

3.1.1.1.1.2 Pilot EPIE 

Pilot EPIE was designed to field-test the system developed for the EPIE Institute. The 
study was intended to implement the short-term goals for operating the EPIE Institute 
as well as gradually incorporating long-term goals. The study was initiated during 1968 
and extended throughout 1969 and 1970. Sixteen objectives were achieved during the 
course of the Pilot EPIE study: data collection instruments and techniques were 
evaluated and revised; observers, interviewers and other information gatherers were 
trained; producer informatio)i was collected; analyst information was collected; user 
information was collected; user interviews were conducted; producer, analyst and user 
information was synthesised; development of software for the special request file was 
completed; the storage and retrieval system was evaluated and updated; broadcast 
information was published in the EPIE Forum; research on patterns of information use 
was conducted; new methods of querying the system and responses were identified and 
established; field agents were trained; limited trial use of the system by a new group of 
subscribers was explored; synthesised information in broadcast, responsive and 
interactive modes was offered; and operation of the customised mode was commenced. 

3.1.1.2 The Operational Phase (1971-1988) 

Once the developmental phase, conducted under Pilot EPIE, had concluded in 
December 1970, the EPIE Institute commenced and expanded formal operations during 
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the 1970s. During this period, the EPIE Institute established a service for collecting, 
analysing and disseminating descriptive information on educational products to all 
sections of the American educational community on a national basis. At the close of this 
decaue, this service had also extended to the western provinces of Canada. 
Today, the EPIE Institute operates from the campus of the Teachers College, Columbia 
University, New York City, where a Program Development and Research Office is 
maintained, whilst a Western Projects Office is located at Kenwood, California, a 
Northeastern Projects Office is situated at Dresden, Maine, and a Midwestern Projects 
Office is situated at Clayton, Missouri. Since early 1988, the EPIE Institute's Software 
Evaluations Office has been located at The Palmer School, C.W. Post Campus, Long 
Island University, Brookville, New York. The major services provided at present by 
these offices of the EPIE Institute are described in the next section. 

3.1.1.2.1 Services 

3.1.1.2.1.1 Outreach 

At the commencement of this period, a foundation grant was provided to promote the 
EPIE Institute's services to schools in the United States. In 1975, 1,500 of the 22,000 
school districts in the United States and Canada were receiving services provided by the 
EPIE Institute (American School Board Journal, 1975). In 1981, the number of school 
districts served by the EPIE Institute had increased to 2,000 (School Library Journal, 
1981). 

3.1.1.2.1.2 Consumers' protection. 

The EPIE Institute was established in 1967 at a time when Ralph Nader and the consumer 
protection movement were a significant influence upon American public life. The major 
aim of the EPIE Institute is to protect educational consumers by exchanging information 
on educational products. Information is gathered from teachers, media specialists, 
students, state departments of education and publishers, as well as from EPIE's extensive 
evaluations of curriculum materials and laboratory tests of educational equipment. This 
information is collected in consumers' reports that are, in turn, made available to EPIE 
members to aid responsible adoption, selection, and implementation of educational 
products. The EPIE Institute also supports more overt action to protect educational 
consumers by acting upon complaints lodged by its membr^rs through its newsletter 
services. These actions might relate to faulty equipment or materials, not receiving 
adequate service from a company, or being subjected to questionable or unethical sales 
strategies. 

3.1.1.2.1.3 Newsletters 

The EPIE Institute currently publishes three newsletters: EPIEgram Materials; 
EPIEgram Equipment; and MICROgram. The newsletters are used for two main 
purposes: to keep school personnel informed on educational consumer issues; and to 
provide feedback information from users to the EPIE staff, who, for instance, may then 
commence action mth a producer to improve a fault in a product. 

3.1 .1.2.1.4 Textbooks 

The EPIE Institute provides evaluations of textbooks in two media: by print through 
EPIE Reports; and on-line through the Textbook PRO/FILE System. The EPIE analysis 
instrument is applied by evaluators to develop four-to-six page analyses of the intents, 
contents, teaching-learning activities, and means of student assessment of all major 
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textbook programs from grades K to 12 in reading, language artr, mathematics, sciences 
and social studies. 

3.1.1.2.1.5 Audio-visual Equipment 

In 1974, the Ford Foundation funded the EPIE Institute to establish a laboratory testing 
program for evaluating all types of audio-visual equipment. Since that date, EPIE 
Reports have been published on cassette recorders, 16mm projectors, cassette 
duplicating equipment, overhead projectors, video-cassette recorders, and sound-slide 
projectors. During this period, the EPIE Institute, in collaboration with the Association 
of Audio-visual Technicians (AAVT), has published a biennially, updated Annotated 
Audiovisual Parts Directory. 

3.1.1.2.1.6 Computer Courseware 

Information on computer co?jrseware available for American schools is reported by the 
EPIE Institute through two media: by print in ITis Educational Software Selector (TESS); 
and on-line through the Microcomputer Courseware PRO/FILE System. 
First published in 1984, TJie Educational Software Selector comprises annotations of both 
courseware (classroom-orientated software) and administrative software. Several 
descriptors are used to annotate each entry: its name; its supplier's catalogue number; 
its type, in terms of instructional or operational mode; its grade level range; its uses, in 
terms of audiences; its scope, by topic and by duration of use; its grouping; by description 
of its operational attributes, its configuration, in terms of hardware-software 
requirements, and its price; its components; its availability; its review citations; and its 
user site contact. 

The EPIE Institute commenced the development of the Microcomputer Courseware 
PRO/FILE System in collaboration with the Microcomputer Center at the Teachers 
College, Columbia University during 1981. The EPIE analysis instrument is applied by 
evaluators to develop two-to-four page, comparative evaluations of all major software 
curriculum packages. The Microcomputer Courseware PRO/FILE System also includes 
evaluations of pre-screened, shorter programs that have met specified criteria for 
courseware. 

3.1.1.2.1.7 Computer Hardware 

Analyses of computer hardware available to American schools are reported on-line by 
the EPIE Institute through the Microcomputer Hardware PRO/FILE System. Analyses 
comprise of four-page, comparative evaluations of the major microcomputer systems and 
peripherals currently being marketed to American schools. The evaluations are based 
upon laboratory testing by the Consumers Union of the U.S., combined with reports from 
school users. 

3.1.1.2.2 Techniques and Procedures 

In order to provide these services to the educational community in North America, 
certain techniques and procedures are applied by the EPIE staff to provide qualitative 
information on educational products. These techniques and procedures are discussed in 
this section. 

3.1.1.2.2.1 Applications to Develop Instructional Materials 

The Educational Products Information Exchange Institute (1980a) has reported on its 
activities to implement procedures for learner verification and revision. These 
procedures encompass gathering and analysing data obtained from field-testing products 
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with appropriate groups of learners and then revising the products on the basis of these 
results. 

Between 1967 and 1971, the EPIE Institute identified that 99 percent of instructional 
materials were not being verified and revised on the basis of data gathered from learners. 
Following congressional testimony by Komoski in 1971, the EPIE Institute has promoted 
the use of learner verification and revision as a m^eans to improve the learning 
effectiveness of educational products. As a result of this testimony, the application of 
learner verification and revision became a particularly controversial issue in American 
education. This arose because of the efforts by the EPIE Institute to effect legislative 
enactments in California in 1972 and Florida in 1974, requiring learner verification and 
revision as a prerequisite for adoption. In response to this p'lblicity, the EPIE Institute 
established a National Learner Verification and Revision task Force in 1974, charged 
with the tasks of defining the process inherent in learner verification and revision and 
developing practical guidelines for its implementation by both publishers and educators. 
The Guidelines for Reporting and Assessing LVR Activities, published as a result of the 
work of the National Learner Verification and Revision Task Force recommended, on 
the part of publishers, that instructional materials should incorporate an instructional 
design, that intended learner options be investigated, that the conditions of use of the 
product should be specified, that techniques for gathering feedback from learners may 
be included, that validation groups used in the learner verification and revision process 
be reported, that the analysis of findings be reported, and thac improvements to the 
material should be made on the basis of these findings. Although few publishers adopted 
these Guidelines, a set of model legislation, derived from the Guidelines, was developed 
by the Lawyers' Committee for Civil Rights Unuer Law and then considered for adoption 
by teachers in Virginia and legislators in Michigan and Maryland during 1975 and 1976. 
At the same time, the legislation enacted in Florida was bolstered by the saioe set of 
Guidelines. 

During the 1975-1976 school year, the Bureau of Audiovisual Instruction of the New York 
City Board of Education conducted a pilot study to gather data in thirty-two school 
districts within New York City for the purpose of determining a policy on learner 
verification and revision. This pilot study, together with the work of educators and 
legislators to implement the Guidelines failed during 1977 because support was not 
forthcoming from publishers. In spite of the failure to implement the Guidelines 
successfully, a survey of twenty-eight junior high school textbooks conducted by the EPIE 
Institute substantiated claims of 2n increasing use of learner verification and revision by 
publishers. 

3.1.1.2.2.2 AppUcations to Select Instructional Matenals 

Selection of instructional materials by committees is advocated by the EPIE Institute. 
Selection committees should be based locally and should consist of administrators, 
teachers, parents, students and other members of the community. Systematic training of 
committee members is viewed as essential. Selection involves determining prospective 
users for particular instructional materials and is governed by the instructional design of 
the materials and the characteristics of the setting in which the materials will be used. 
Sequential steps to be taken by selection committees are to review and to examine 
available materials within a field of interest through the use of checklists and rating scales. 
Then, selertion committees screen these materials by mtv.ns of appraisal forms based 
upon co-operatively agreed-upon criteria related to both the materials and the 
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appropriate instructional setting. Komoski (1986) reported that "recently, EPIE has 
added the use of computer-generated curriculum alignment reports as an important 
element to be used in the screening process". On the basis of this screening, selection or 
production of materials can be made within a 'decision arena' of five alternative courses 
used alone or in combination: continued use of existing materials within existing 
programs; selection of materials on the basis of learner and teacher characteristics and 
approach to instruction; development of materials locally, regionally, or at a state level; 
initiation of inservice training of personnel in the use of materials and implementation 
of programs; and initiation of broader curriculum development for the appropriate 
programs. 

3.1.1.2.2.3 Applications to Evaluate Instructional Materials 

The instruments used by the EPIE Institute to evaluate educational products are 
adaptations of an original instrument published by Eash (1972b). Eash's instrument 
contained five sections: I Objectives; II Organisations; III Methodology; IV Evaluation- 
and V Comment. 

Following adoption by the EPIE Institute of Eash's instrument, Elliott (1985) has 
reported that the first version of EPIE form A was developed in response to feedback 
from participants at a workshop in which Eash assisted the EPIE Institute train teachers 
m California to use his instrument to analyse textbook programs in reading. The 
fundamental alterations that occurred in the transition from Eash's instrument to EPIE 
form A are best related in Elliott's words. 

The main issue that led to the feedback and the revision focused o.i whether EPIE analyses 
should favor some specific instructional design provisions over others or simply describe the 
orovisions made m each set of materials and leave it to the selector to express preferences. In 
the Eash instrument with which we started the training at Los Angeles, analysts were asked to 
rate a number of instructional design features on a scale of ten (e.g., fiilly stated 'behavioral' 
objectives were given the highest rating and very general outcOitie statements the lowest). In 
the EPIE form A version that emerged from these sessions, analysts were asked to describe 
each instructional design provision as precisely as possible (e.g.. Objectives give [check aU that 
apply]: a. expected bebavior(s), b. conditions under which it/they should occur, c. performance 
standard, d. other ). 

The original Eash instrument was based on a single point of view about what constitutes good 
mstruction; EPIE form A allowed for alternative views and stressed making a good match be- 
tween: (1) user needs and preferences and (2) one or more of a number of different approaches 
uUilt into the sets of materials available, on the market...." 
A significant feature of this instrument has been its capacity for adaptation to different 
educational contexts and for revision, based upon criticisms received from educators who 
use EPIE Reports. For instance, during 1984, major revisions were undertaken to the 
version of the EPIE analysis instrument applied to the analysis of textbooks (Educational 
Products Information Exchange Institute, 1985). This revised version comprises four 
sections: Contents (scope, content organisation, and other content considerations)- 
Methodology (typical lesson/learning approach, levels/types of thinking in learning 
activities, provision for extension/enrichment activities and comment on methodology); 
Tests and Assessment (descri|)tion of provisions, comments on tests and assessment); 
and Other Considerations (program implementation, technical quality of program 
materials, summary and goodness of fit, analyst's summary comment). As a result of these 
changes, Elliott indicated that "future EPIE Reports will contain more critical comments 
about such matters as the clarity of learning activity instructions and the 'considerateness' 




38 



of the text narrative, while maintaining neutrality concerning instructional approaches or 
philosophies". 

Several significant features have been incorporated within the EPIE analysis instrument: 
instructional design analysis; identification of biases, such as racism and sexism, in 
instructional materials; and the matching of readat ' ity levels of instructional materials 
to students' reading levels. 

3.1.1.2.2.3.1 Instructional design analysis 

An instrument must be based upon a model of curriculum development in order to 
provide valid analyses of curriculum materials. Such an application is termed 
instructional design analysis. When the EPIE analysis instrument is applied to analyse a 
curriculum material, qualitative information is elicited for each of the constructs or 
elements of the curriculum in terms of Tyler's objectives model: the intents; the contents; 
the teaching-learning methodology; and the means of student assessment. 
By applying instructional design analysis to curriculum materials, an analyst uses EPIE 
analysis instrument to perform several functions: descriptive analysis, which concentrates 
upon elucidating the rationale and structure; evaluation, which provides the capability to 
judge curriculum materials against a range of criteria; and decision-making, which 
provides judgements allowing users to select and to implement curriculum materials. 
Once the instructional design analysis of a curriculum material has been determined, an 
analyst is in a position to make professional judgements on questions of 'goodness of fit' 
or congruence. 'Goodness of fit' or congruence is used to apply the results of descriptive 
analysis and evaluation to determine the characteristics of both selection and 
implementation types of the decision-making function and, therefore, is the main purpose 
of conducting "nstructional design analysis of materials and programs. The two forms of 
congruence - internal and external - are now discussed. 

3.1.1.2.2.3.1.1 Internal congruence 

Internal congruence refers to the degrees of congruence between the four constructs of 
the curriculum and among the constituents of each construct. In the first instance, an 
analyst applying the EPIE analysis instrument, will ask questions, such as: Does the scope 
of the contents fulfil the developer's expressed or implied purpose, and explicitly stated 
aims and objectives? In the second instance, the analyst will ask: Are the goals and 
objectives consistent with the developer's expressed purpose? Statements based upon 
answers to such que' ions provide information on the integrity of the instructional design 
within a curriculum material. 

3.1.1.2.2.3.1.2 External congruence 

External congruence refers to a measure of the 'goodness of fit' between the constructs 
of the curriculum and the environment in which the curriculum material will be 
implemented. The environment includes the educational program, the intended learners, 
the requirements of the teacher and the requirements of the learner. If the instructional 
designs of both a curriculum material and an intended instructional program have been 
analysed in the same way, a decision-maker can base such 'goodness of fit' upon a 
judgement of how well a material is likely to match a local pn gram, if forms of 
learner-based verification (pilot-testing and field-testing) are unavailable. 

3.1.1.2.2.3.2 Identification of biases 
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In collaboration with the Council on Interracial Books for Children (CIBC), New York 
City, the EPIE Institute conducted a research project during 1974 to determine the 
characteristics of racist biases in instructional materials and to develop criteria for 
investigating racist biases in instructional materials (Educational Products Information 
Exchange Institute, 1975: 27-54). At the same time, the EPIE Institute contracted 
Women on Words and Images (WOWI). Princeton, New Jersey, to conduct research on 
the characteristics of sexist biases in instructional materials and to develop criteria for 
investigating sexist biases in instpjctional materials (Educational Products Information 
Exchange, 1975: 55-84). On the basis of this research, criteria have been incorporated 
within the EPIE analysis instrument for evaluators to report both racist and sexist biases. 

3.1.1.2.2.3.3 Matching readability levels of instructional materials to 
students' reading levels 

An instrument, the Degrees of Reading Power, based upon the Bormuth readability 
formula, has been developed by the New York State Department of Education (1980). 
The Degrees of Reading Power can be employed for diagnostic assessments of both 
students and instructional materials. Three main concepts are employed in the Degrees 
of Reading Power: the assessment of students' reading abilities through the use of cloze 
passages; the assessment of the readability of materials; and the computer-based 
matching of materials of appropriate difficulty to students. 

Komoski (1982) reported that the Degrees of Reading Power was first employed for these 
purposes by the EPIE Institute during January 1982 at a group of elementary schools in 
New York City and on Long Island. Since 1984, these measures have been incorporated 
into evaluations of textbooks reported in the Textbook PRO/FILE System. 

3.1.1.2.2.4 Applications for Curriculum Alignment 

Curriculum alignment is based upon a principle that student achievements can be 
improved by aligning the objectives, instruction through both materials and practices, and 
assessment. Developed through a collaborative project between the Los Angeles Unified 
School District and the Southwest Regional Laboratory for Educational Research and 
Development, Los Alamitos, California (Niedermeyer and Yelon,1981; Scott, 1987), 
curriculum alignment represents an effective way to match the use of different types of 
educational products - textbooks, supplementary materials, tests, computer courseware 
and videotapes - to the elements of the instructional design in the curriculum. 
During 1985, the EPIE Institute developed and piloted a comparable service in 
curriculum alignment by modifying and extending an earlier program: the TextTest 
Matching Service. The Curriculum Alignment Services for Educators (CASE) provides 
information in a database, the Integrated Instructional Information Resource (IIIR), on 
textbooks, audiovisual materials, computer software, supplementary materials, and both 
norm-referenced and criterion-referenced tests, which can be aligned with a school's 
curriculum objectives. As well as allovidng teachers to select those materials that are 
aligned to their programs in a particular subject area, the Curriculum Aligmnent Services 
for Educators also allows teachers to check the potential non-alignment of educational 
products. 

Komoski (1986) provided the following description of the Integrated Instructional 
Information Resource database and intended, future developments in its 
implementation. 

The information structure of the database begins with the identification of a broad range of 
curriculum descriptors (topics, processes, attitudes etc.) for a given curriculum area. These 
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descriptors are deliberately wide enough in range to enable us to describe and analyse very 
traditional curriculum materials, and also materials that might be considered by some to be 
quite avant-garde. In fact, some descriptors, based on the work of leading edge thmkers in 
curriculum go beyond any materials currently available to schools. In short, this universe of 
curriculum descriptors may be thought of as a kind of social document that describes where 
curriculum has been, where it is today, and where it might be going m the future. As a result, 
the IIIR Database may be used for designing and updating curricula as well as analysmg cur- 
rent curricula... 

...Our next step in developing the potential of the IIIR Database for teachers' use will be to 
complement the IIIR Database with a modular instructional management system that teachers 
may use to draw on the IIIR Database for a variety of teaching tasks. These modules will enable 
teachers to use the database to design lessons, record student progress, and assess the differen- 
tial effectiveness of specific teaching strategies and materials. 

A number of state education agencies that have been using the IIIR Database agree with us 
that the database, when complemented with instructional management for teacher use, will be- 
come an extremely important tool through which teachers' instructional skills may be honed". 
Funded by grants from the Carnegie Corporation of New York, the Ford Foundation, 
the Richard Lounsbury Foundation and supported by the Council of Chief State School 
Officers, both the Integrated Instructional Information Resource and the Curriculum 
Alignment Services for Educators are being implemented at present in school districts 
in different parts of the United States. In 1986, services were commenced in mathematics 
and science for grades K to 8, followed in 1987 by services in language arts and reading. 
By the end of 1989, the Integrated Instructional Information Resource will be operating 
for mathematics, science, language arts, reading and social studies. 
Komoski (1987) has elaborated upon his description with statements on how the 
Integrated Instructional Information Resource can affect five basic principles of 
curriculum and instruction derived from questions Tyler (1949) stated to define his 
framework. 

1. What educational purposes should a school seek to attain? 

2. What educational experiences can be provided that are likely to attain these purposes? 

3. How can these educational experiences be effectively organised? 

4. How can we determine whether these educational purposes are being attained? 

5. How can a school maintain a systemic balance among the activities called for by the 
first four questions? 

Komoski states that the Integrated Instructional Information Resource addresses the first 
basic principle by providing a set of curriculum descriptors to assist in: 

1}uilding locally developed curricula. By using this adaptable set of descriptors on a special 
curriculum design spreadsheet, curriculum committees can explore curriculum 'what-ifs' and 
continually order and reorder a school's curriculum, subject area by subject area, grade by 
grade; 

analyzing, correlating, and comparing the subject matter content, and the cognitive proces- 
ses embedded in textbooks, other learning materials, and tests to the content and processes 
called for in a school's curriculum; 

documenting and tracking the evolution of curriculum thinking and practice over time within 
a district, a state, or across states; and 

using state and nationally recommended curriculum standards to inform local curriculum 
development. For example, the Resource's descriptors have been used to code state cur- 
riculum guides and such national-level standards as those of the National Council of Teachers 
of Mathematics. This means that schools can use the Resource as a means of informing their 
curriculum work with these state and national efforts to improve curriculum planning." 

(Komoski, P.K., 1987, Educational Technology: Vie Closing-in or the Opening-out of Cur- 
riculum and Instruction^ Syracuse: ERIC Clearinghouse on Information Resources, 27-28) 




The Integrated Instructional Information Resource also provides information related to 
the second and third pronciples of Tyler's framework, described by Komoski as: 

"information on mediated learning experiences ranging from textbooks and the proliferating 
array of other instructional materials (computer-based, video-based, and print-based), to the 
increasing numbers of integrated systems, some of which combine computer-aided instruction 
and management with print and other media; 

information about nonmaterials-based learning* experiences and about the ways teachers can 
organise use of materials to go beyond their obvious uses. This function includes such teacher- 
generated strategies, student studies of nature, local government, their own behavior, as well 
as having students carry out useful projects with their school and local community ..." 

(Komoski, P.K., 1987, Educational Technology: Vie Closing-in or the Opening-out of Cur- 
ricuhtm and Instruction y Syracuse: ERIC Clearinghouse on Information Resources, 28) 
Komoski describes information in the Integrated Instructional Information Resource 
that relates to the fourth of Tyler's basic principles, as follows. 

"By using the Resource's curriculum descriptors to analyze and code test objectives and/or 
items into the Resource's database on tests, information about relevant norm-referenced and 
criterion-referenced tests maybe accessed, correlated, and aligned with a school's curriculum 
goals, its materials, and its nonmaterials-based teacher strategies. It is also possible to create 
a link between the Resource and any of the many available banks of criterion-referenced test 
items." 

(Komoski, P.K., 1987, Educational Technology: The Closing-in or the Opening-out of Cur- 
riculum and Instruction^ Syracuse: ERIC Clearinghouse on Information Resources, 29) 
In explaining the relationship between the fifth basic principle and the Integrated 
Instructional InT-^rmation Resource, Komoski states that: 

"A major purpose of the Resource is to enable schools to expedite the time-consuming, sys- 
tematic analysis and correlation that are essential to curriculum planning, materials and test 
selection, and curriculum alignment. As a result, it makes it possible for a school's curriculum 
planners and teachers to devote more time to thinking through the systemic concerns involved 
in using the results of those analyses and correlations in applying the crucially important fifth 
principle of curriculum and instruction." 

(Komoski, P.K., 1987, Educational Technology: The Closing-in or the Opening-out of Cur- 
riculum and Instruction^ Syracuse: ERIC Clearinghouse on Information Resources, 30) 
Finally, two additional goals of the Integrated Instructional Information Resource are 
specified by Komoski (1987). Firstly, the Resource can be applied as a means of training 
teachers to select learning experiences for students using a combination of commercial 
materials together with teacher and student generated learning strategies. Secondly, 
research at the EPIE Institute is also focusing upon determining ways that parents can 
use the Resource to become informed on matters related to the curriculum so that they 
can support their children's learning at home. 

3.1.1.2.2.5 Applications for Utilisation of Instructional Materials 

To counteract what are now viewed to be both inadequate procedures and the widespread 
use of dubious practices for textbook adoptions in the United States, the Educational 
Products Information Exchange Institute (1986a) has developed and is promoting a 
utilisation policy which extends the process of adoption beyond selection of instructional 
materials to their use in the classroom. This utilisation policy is based upon three 
assumptions: firstly, that most textbooks are inadequate to promote positive classroom 
instruction; secondly, that most supplementary materials are flawed; and thirdly, that 
teachers require support, training, monitoring and communication with colleagues on 
matching instructional materials to the capabilities of individual students. The utilisation 
process includes adoption as an initial stage, which involves both selective and evaluative 
aspects. This is then follov/ed by in-service training provided by publishers on the 
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technical aspects of an educational product and its management system. At the same time, 
the comprehensibility, the readability and the content appropriateness of the product 
should be considered independently from recommendations by publishers. In these ways, 
utilisation combines several techniques already mentioned within a single policy. 

3.1.1.2.2.6 Research related to the Uses of Educational Products 

Two significant studies conducted by the EPIE Institute have researched the use of 
educational products in classrooms of American schools (Educational Products 
Information Exchange Institute, 1977a; Komoski, 1985). The first study, named the 
National Survey and Assessment of Instructional Materials (NSAIM), was funded by the 
Lilly Endowment during 1974, In 1974 and 1975, the EPIE Institute gathered baseline 
data from a nationally stratified sample of more than 12,000 classroom teachers on the 
nature ani quality of instructional materials. It was found that instructional materials are 
used between 90 per cent and 95 per cent of instructional time, which includes the use of 
textbooks for 70 per cent of instructional time. Research also investigated the degree to 
which learners' needs are matched to appropriate instructional materials. It was reported 
that only about half of the 12,000 teachers surveyed were involved in selecting the 
instructional materials they used. 

In the second study conducted by the EPIE Institute for the National Institute of 
Education, it was found that instructional materials used in classrooms did not match 
learners' needs, contrary to the assertions of administrators and teachers. A one-group 
pretest-posttest design was administered in a number of schools in both affluent and 
economically depressed communities. The findings pointed to most students from all 
socio-economic backgrounds knowing a considerable proportion of the subject matter of 
instructional materials before their use and to few students making substantial gains in 
achievement tests administered after the use of the instructional materials. 

3.1.1.2.2.7 Applications to Train Analysts of Educational Products 

Since its foundation, the EPIE Institute has promoted an extensive and stringent program 
to train the analysts it employs to evaluate educational products. This is achieved through 
the use of training materials and a process, each of which is now described. 
EPIE training Form I was the first material published (Educational Products Information 
Exchange Institute, 1977b). This instrument is designed for either class use or 
self-instruction. The instrument is a variant of EPIE analysis instrument, which was being 
used at that time to analyse educational products. EPIE training Form I comprises the 
following parts: I Product Identification and Background; II Instructional Design 
Constructs (A. The First Instructional Design Construct: Intents, B. The Second 
Instructional Design Construct: Contents, C. The Third Instructional Design Construct: 
Methodology, D.The Fourth Instructional Design Construct: Means of Evaluation); III 
Instructional Design Fit; and IV Other Considerations (Content Authenticity: Accuracy, 
Fairness, and Currency). EPIE training Form I specifies a set of common criteria on 
which trainees must base their descriptive, analytical and evaluative comments about a 
material. Instructions are supplied explaining each step in the analytic process, and 
examples of statements for each design construct are appended. 
Recently, the Educational Products Information Exchange Institute (n.d.) has developed 
a module for use in providing educators with the knowledge and skills to evaluate all types 
of educational materials. Two ways for determining analysis of materials are presented 
in the module: firstly, appl' ^^tion of four curriculum constructs - intents, contents, 
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methodology and evaluation - to provide qualitative analysis of materials; and secondly, 
correlating concepts, textbooks, supplementary instructional materials, computerised 
software programs, films, videotapes and tests to provide quantitative analysis of 
materials. 

Qualitative analysis is presented through demonstration and discussion of six concepts: 
ALeaming Materials Continuum; Ralph Tyler's Rationale; EPIE Institute's Curriculum 
Analysis Framework; A Bridge 'Analogy' of the Curriculum and Instruction Process; 
Internal and External Curriculum Congruence; and Development of an Integrated 
Instructional Information Resource. Quantitative analysis focuses upon matching 
concepts and educational materials in Curriculum/Content/Evaluation Correlation, and 
linking this process to the Integrated Instructional Information Resour' and its 
application in the Curriculum Alignment Services for Educators. 
The process that is employed in this training program can be conveyed by way of an 
illustrrtion. The Educational Products Information Exchange Institute (1984) provides 
an account of the rigorous training program implemented for developing the 
Microcomputer Courseware PRO/FILES. 

"In 1982 as the flow of microcomputer products bemg marketed for school and home use 
began to mcrease, EPIE (working mih Consumers Union and funded by the Ford and Car- 
negie Foundations) joined with the Urban Superintendents Technology Consortium to tram 
and certify educators to evaluate microcomputer courseware. Some of the geographic areas 
mvolved in evaluation are: Albuquerque, Boston, Cmcinnati, Detroit, Houston, SaJr Lal:e City, 
Florida's Broward County, Alberta, San Francisco Bay Area, and the Universitj^ of Southern 
Alabama... 

...Centered at EPIE's Program Development and Research Office, Teachers College, Colum- 
bia University, the courseware evaluation process begins with extensive training and then cer- 
tification of prospective analysts. These prospective analysts, all with relevant backgrounds m 
computers, education, and specific content areas, are trained to use EPIE's evaluation instru- 
ment and to practice its application on a number of courseware programs. Once trained, 
prospective analysts may apply for 'certification'; each must complete an analysis of a desig- 
nated piece of courseware which is then compared to a 'model' analysis by a three-person 
evaluator certification committee. Applicants not meetmg a high standard of expertise are 
refiised certification. This process ensures that EPIE uses only highly skilled and motivated 
analysts, which m turn then helps ensure that our Courseware PRO/FILES are of the highest 
quality. 

Once analysts have been certified, their progress is closely monitored; each is paired with an 
experienced analyst and receives both formal and informal training. High mter-rater reliability 
attests to the efficacy of this approach to training analysts." 

(Educational Products Information Exchange Institute, 1984, TIte Educational Software 
Selector, 547-548). 

3.1.1.2.2.8 Applications to Teacher Education 

The EPIE Institute has been active in teacher education since 1973-1974, when a program 
to train teachers in analysing instructional materials was introduced in Pennsylvania and 
California, and was later extended to other states and Canada. 
The EPIE Institute has developed two sets of modules for training teachers to select and 
effectively use curriculum materials. The first set to be developed, the Packaged Training 
Workshop in Instructional Materials Selection, consists of thirty modules developed 
between 1978 and 1979. The second set comprises ten modules developed during the 
conduct of the Teacher Information Exchange (TIE) project between 1980 and 1982. 
Funded by the National Institute of Education, the development, field-testing and 
revision of the Packaged Training Workshop in Instructional Materials Selection, are 
reported by the Educational Products Information Exchange Institute (1980b). The 
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thirty r. odules consist of three main groups: sixteen modules, the basic components of 
the set, developed, field-tested and revised during 1978-1979; four modules adapted 
from the first group for use with special educators, developed and later field-tested b/ 
the Wayne County Public Schools, Mith?gan during 1979; and ten modules developed 
during 1979 following empirical research conducted by the EPIE Institute to gather and 
to analyse data that indicated widespread and gross misfitting of curriculum materials to 
the capabilities of students. A project to disseminate the modules and train teachers was 
initially conducted in Illinois during 1979 and later, in 1980, extended to other states. 
In 1980, the Teacher Information Exchange (TIE) project, reported by the Educational 
Products Information Exchange Institute (1986b), was implemented for two years in a 
dozen elementary and junior high schools in New York City. The EPIE Institute's staff 
worked with teachers through classroom observations, consultations, workshops and 
meetings to identify issues related to the use of curriculum materials in classrooms. The 
same teachers were provided with in-service training about how to use curriculum 
materials more effectively once they had been selected. An important feature of this 
project was to train these teachers sufficiently so that they could share their training with 
other teachers. To facilitate teacher education, ten modules v;ere developed by the EPIE 
Institute and field-tested in the participating schools. These modules addressed the 
following issues: 1. an overview of the concept of time-on-task; 2. and 3. educational 
objectives; 4. and 5. a management system involving tests and record-keeping devices; 
6. and 7. supplementary materials, their adaptation, and planning of worksheets; 8. 
classroom structure as related to using curriculum materials and learners' time-on-task; 
9. pacing lessons; and 10. instituting routine in using instructional materials. 
Finally, the Educational Pr^ ^ucts Information Exchange Institute (1986c) has developed 
a set of modules to facilitate the training of teachers in relation to the implementation of 
the Degrees of Reading Power in school districts. The modules are intended to be used 
at either EPIE training workshops or school-based workshops. 

3.1.2. The Social Science Education Consortium (SSEC) 

The Social Science Education Consortium (SSEC) was established during 1963-1964 at 
the U.iiversity of Colorado, Boulder, Colorado. Its aims are to collect and disseminate 
materials for social studies education, to support development and implementation of 
new social studies materials and to improve working relationships between personnel in 
various social studies education projects. 

3.1.2.1 The Procedure for Selecting Curriculum Materials 

Davis and Eckenrod (1972) provide an account of the procedure recommended by the 
Social Science Education Consortium for selecting curriculum materials for social 
studies. Two major steps are involved in this process: firstly, a statement of broad 
program goals, such as the guidelines developed by the National Council for the Social 
Studies, can be used to identify available curriculum materials; and secondly, evaluation 
of curriculum materials that appear to support the program goals by use of the Social 
Science Education Consortium's Curriculum Materials Analysis System. 

3.1.2.2 The Curriculum Materials Analysis System 

Morrissett et al. (1968) reported the development of an instrument to evaluate 
curriculum materials. The instrument originated informally as a brief form containing a 
dozen or so questions. This Corm was revised and enlarged on several occasions but was 
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first formally applied as part of activities undertaken with the Wabash Valley Education 
Center, Indiana, early in 1966. The original version of this instrument, published by 
Morrissett and Stevens (1967), comprised the following sections: 1.0 Descriptive 
Characteristics; 2.0 Rationale and Objectives; 3.0 Antecedent Conditions; 4.0 Content; 
5.0 Instructional Theory and Teaching Strategies; and 6.0 Overall Judgements. No sooner 
had this original version of the Curriculum Materials Analysis System been published ',han 
an initial, revised version was published (Stevens and Morrissett, 1967-1968; and Stevens 
and Fetsko, 1968). This version was the result of reworkings conducted at a conference 
sponsored by the EPIE Institute at Lake Mohonk, New York in 1966 and at Purdue 
University in April 1967. Further reworkings of the instrument also occurred as a result 
of a conference held at the University of Colorado in May 1968, and as a result of 
criticisms and suggestions for revision contributed by Charies Adair, Frances Klein 
Michael Scriven, Hilda Taba and Louise Tyler. In May 1971, a second revised version of 
the mstrument (Social Science Education Consortium, 1971) was published, containing 
short, intermediate and long forms and including two additional sections. This version 
is arranged as follows: 1.0 Product Characteristics; 2.0 Rationale and Objectives; 3.0 
Content; 4.0 Theory and Strategies; 5.0 Antecedent Conditions; 6.0 Evaluation;' 7.0 
Background of Materials Development; and 8.0 Background of the .Analysis. Singleton 
(1987) reported that the Curriculum Materials Analysis System has not been adapted to a 
computer-based program. 

Analyses of social studies curriculum materials undertaken by the Social Science 
Education Consortium, using a two-page framework derived from the Curriculum 
Materials Analysis System., are published in successive editions of the Social Studies 
Curriculum Materials Data Book. 

3.1.2.3 Uses of the Instrument 

The authors cite eight possible uses of this instrument (Morrissett et al.): general library 
use; analysis of trends within curriculum materials; field data collection about classroom 
use of curriculum materials; decision-making in the selection of new curriculum 
materials; provision of analyses of curriculum materials in terms of a curriculum model; 
promotion of all dimensions of curriculum development; introduction of new ideas and 
approaches in curriculum materials through in-service education; and acquainting 
preservice teacher trainees with the range of curriculum materials and the ability to 
perform their own analyses. Application of the Curriculum Materials Analysis System to 
several of these uses has been reported by Morrissett et al. and Davis and Eckenrod. 

3.1.3 The Council on Interracial Books for Children (CIBC) 

The Council on Interracial Books for Children (CIBC) was founded in 1966 by writers, 
editors, illustrators, teachers and parents to promote literature for children that better 
reflects multi-acial and multicultural values. The CIBC is located at New York City. 
The program of the CIBC consists of three main activities. Conferences and workshops 
organised by resource specialists and scholars representing Blacks, Puerto Ricans, 
Chicanos, Asian Americans, Native Americans and women, focus parents, publishers and 
educational professionals on biases found in textbooks and on developing criteria for 
analysing instructional materials. A Third Wodd Writers Contest promotes the 
anti-racist and anti-sexist works of previously unpublished writers. The Interracial Books 
for Children Bulletin provides evaluations of racist and sexist biases in instnictional 
materials. In 1975, the Carnegie Corporation provided a grant to the CIBC to establish 
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a Racism and Sexism Resource Center for Educators. Its staff develops anti-racist and 
anti-sexist curriculum materials, lesson plans and teaching strategies, provides 
consultants and conducts teacher training programs. 

The Council on Interracial Books for Children (1977) published a content analysis 
instrument for assessing the portrayals of women and ethnic groups - African Americans, 
Asian Americans, Chicanes, Native Americans and Puerto Ricans - in U.S. history 
textbooks. In excerpts describing each of these groups, a list of facts is provided that 
reflects contents thought to be absent in biased textbooks. The evaluator examines a 
textbook of interest to determine whether each fact is present. Using a simple numerical 
procedure, based upon the quantity and the quality of coverage, the evaluator completes 
a rating scale with the following categories: whether the text provided incorrect 
information; whether the text pro aded no information; whether the text failed to deal 
with the historical period from which the fact was derived; whether the text provided 
limited information; or whether the text provided full information. A text is scored by 
computing numerical values. Texts with the highest scores are considered to provide a 
more balanced treatment of targeted groups. 

Garcia and Armstrong (1979) have identified two limitations of the procedure employed 
by the CIBC in that publication. Firstly, the lack of clear decision rules for evaluators to 
distinguish between the categories may lead to low inter-rater reliability. Secondly, 
because the underlying historical perspective only accepts a simple interpretation, a 
'colonial model', in which whites always oppress blacks and men always oppress women, 
this perspective, they believe, is as slanted as the one it is proposed to redress. 
Late in 1979, the United States Office of Education invited the Council on Interracial 
Books for Children to present a set of bias-free guidelines to program officers at the 
United States Department of Health, Education and Welfare. These guidelines were 
later published (Council on Interracial Books for Children, 1979). In this publication, 
the Council on Interracial Books for Children compiled guidelines originally presented 
in its earlier publications as well as from other sources. Within this publication, the 
attributes of racist and sexist biases in both storybooks and textbooks are described, 
guidelines are presented for specific minority groups, and checklists of criteria that relate 
to each group are appended. The Council on Interracial Books for Children (1980) 
excerpted ten guidelines to identify racist and sexist biases in instructional materials: to 
check illustrations; to check story lines; to examine lifestyles; to weigh the relationships 
between people; to identify the heroes; to consider the author's and the illustrator's 
backgrounds; to check out the author's perspective; to watch for loaded words; and to 
look at the copyright date. 

3.1.4 Women on Words and Images (WOWl) 

Founded in 1970 by Joan Bartl, Rogie Bender, Pryde Brown, Cynthia Eaton, Carol Jacobs 
and Ann Stefan, the Women on Words and Images (WOWI) at Princeton, New Jersey is 
a group of researchers and consultarUs on sexism in education. 
Women on Words and Images (1972) published the results of two years' research into 
sexism in 2760 stories in 134 readers from 14 American publishers. The results indicated 
that sexism is both evident statistically and through content analysis in the 
representations, depictions and occupations of males and females in the sample of 
readers analysed. A slide show was then developed io present these findings at 
workshops. 
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As a result of its contribution to the EPIE Institute's analysis of instructional materials 
for career education, another publication by Women on Words and Images (n.d.a) 
analysed sex-role stereotyping in 100 nationally distributed career education materials. 
Aslide show, based on this publication, was also produced. Women on Words and Images 
(n.d.b) has also produced a slide show on the results of a content analysis of sex-role 
stereotyping in 25 foreign language textbooks by 16 publishers. 
Women on Words and Images (1979) published a checklist for evaluating sexism in 
instructional materials. Evaluators should answer four questions when using this 
checklist: whether equivalent titles of address are used in the material for both men and 
women; whether irrelevant information about a woman's family is included when 
describing her role at work; whether achievements of both males and females are cited; 
and lyhether a woman is described in terms of physical appearance whilst men are 
described in terms of accomplishments or character. 

3.2 A Survey of Other Educational Organisations 

In order to survey comprehensively the actions of other American educational 
organisations through their publications , the author conducted a review of research 
through a systematic search of the annotated index of the Resources in Education (RIE) 
database compiled by the Educational Resources Information Center (ERIC) 
Washington, D.C. The conduct of this search has been previously reported (Watt, 1987). 
The purpose of the search was twofold: firstly, to identify documents on selecting and 
evaluating educational products published by national educational organisations; and 
secondly, to identify guidelines for selecting and evaluating educational products 
published by state education departments and school districts. 

3.2.1 Method 

The search of the annotated index in the Resources in Education database, was conducted 
manually and included the period between the estaolishment of the database in 
November 1966 and December 1985. This search was conducted through the Subject 
Index using the following descriptors: bilingual education; instructional materials; 
multicultural education; multicultural textbooks; reading material selection; textbook 
bias; textbook evaluation; and textbook selection. 

Because the original purpose of the search had been to identify documents that relate to 
generically-based research and to research related to bilingual-bicultural, multicultural 
education and related areas, research that was based in other subject areas was excluded. 
Initially, identification was made on the basis of the relevance of each document's tide^ 
and then, in each case, recourse was taken to examining relevant abstracts indexed in the 
Main Entry section. It was possible to accept or reject each document on the basis of the 
description provided in the abstract satisfying a specific criterion. This criterion specified 
that the document included subject matter that related substantially to the selection and 
the evaluation of curriculum materials, including evaluation of bias. 

3.2.2 The Results 

Annotations of rt search on selecting and evaluating educational products that have been 
entered in the Resources in Education database, represent an infinitesimal proponion of 
its total entries. The 158 documents identified, represent 0.06% of a total of 250,173 
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documents (ED 010 000 through ED 260 172) indexed in the Resources in Education 
database during this period. Among the 158 documents, 70 related to research about 
general developments in selecting and evaluating curriculum materials, and 88 related to 
research about selecting and evaluating curriculum materials for bilingual-bicultural 
education, multicultural education and related areas. One hundred and forty-seven 
(93.0%) of the contributions in this field to the Resources in Education database, 
originated from sources in the United States of America. 

3.2.3 Discussion 

3.2.3.1 Universities 

Of the 147 documents, twenty documents (13.6%) originated from universities. Of the 
sixteen universities contributing research, three documents originated from the City 
University of New York (Cohen and Koehler, 1975; Eash 1969; and Eash 1970) and two 
documents each originated from Tlie Ohio State University (1976; and 1977) and the 
Michigan State University (Levine 1969; and Ward, 1969). Single documents were 
contributed by authors of other universities: Hernandez and Melnick (n.d.) at Hartford 
University; the George Washington University (1968); Singh and Barnard (1969) at the 
University of South Florida; Mehlinger and Patrick (1970) at Indiana University; 
McGuigan (1971) at the North Carolina State University; Wilcox (1971) at the University 
of Southern California; Doiima (1973) at the University of Michigan; McKeon (1975) at 
The State University of New Jersey; Williams et al. (1976) at the San Diego State 
University; Rabin (1978) at the University of Pennyslvania; Harrison (1979) at the 
University of Arizona, Keith (1981) at Stanford University; and Ventura (1983) at the 
University of Kansas. 

Seven universities were represented in eight contributions (5.4%) to joint research. Two 
contributions to joint research were made by the Northern Illinois University (Simpson 
and Loveall, 1976; and Charuhas, 1984). Authors from the remaining universities 
contributed single documents: the Educational Products Information Exchange Institute 
et al. (1969) for the New York State University; Armstrong (1973) for Wisconsin 
University; Magisos (1973) for The Ohio State University; the Alabama University and 
Mobile County Public Schools (1976); Tierney et al. (1980) for the Illinois University; 
and McGrew (1983) for the University of Northern-Iowa. 

Both academic research and higher degree theses were represented in documents 
contributed by tertiary institutions. 

3.2.3.2 Educational Research Organisations 

Twenty documents (13.6%) originated from educational research organisations. Of this 
contribution, four documents originated from the Educational Products Information 
Exchange Institute (1975, 1976a, 1976b, 1976c), four documents originated from the 
Educational Testing Service (Epstein et al, 1971; Walton, 1973; Walton et al, 1973; and 
Eash et al, 1975), and four documents originated from the Social Science Education 
Consortium (Knight and Hodges, 1970; Knight et al., 1971a; Knight et al., 1971b; and 
Haley, 1982). Three documents were contributed by regional educational laboratories, 
of which the Far West Laboratory for Educational Research and Development 
contributed two (Banathy et al., 1976; and Far West Laboratory for Educational 
Research and Development, 1980) and the Central Midwestern Regional Educational 
Laboratory contributed one (Tom, 1977). The five remaining documents originated from 
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the Institute of Educational Development (1969), the Educational Research Service 
(Kunder, 1976), the School Information and Research Service (Orlich, 1979), the South 
Carolina Vocational Education Research Co-ordinating Unit (Reynolds, 1981), and the 
Education Development Center (Cotera, 1982). 

In addition, the Educational Products Information Exchange Institute (1969; and 1974) 
collaborated with.other organisations to develop two other documents (1.4%). 

3.2.3.3 Professional Associations 

Professional associations were responsible for two types of documents: firstly, documents 
published by those associations; and secondly, conference papers sponsored by those 
associations. Both national and state professional associations are considered in this 
discussion. 

Ten professional associations were responsible for contributing fourteen documents 
(6.8%) to the Resources in Education database. Of this contribution, four documents 
originated from the National Council of Teachers of English (Perkins, 1967; National 
Council of Teachers of English, 1970; Perkins, 1972a; and Perkins, 1972b) and two 
documents originated from the National Education Association (1973; and 1978). Single 
documents were contributed on behalf of professional associations by different authors: 
Dusel (1970) for the Foreign Language Association of Greater Sacramento; Deya (1975) 
for the Southern Conference on Language Teaching; Williams (1975) for the Michigan 
Education Association; Cohen (1976) for the Association for Childhood Education 
International; Witter (1977) for the Oregon Association for Supervision and Curriculum 
Development; Klein (1978) for the Association for Supervision and Curriculum 
Development; Johnson (1979) for the American Council on the Teaching of Foreign 
Languages; and Haas (1985) for the National Council for the Social Studies. 
Furthermore, six professional associations collaborated to publish the remaining five 
documents (3.4%). These collaborations were undertaken by the Joint Committee of 
the National Education Association and Association of American Publishers (1972). 
Kamhi (1982a; 1982b; and 1982c) who produced three documents in a collaborative 
effort for the American Library Association, the Association for Srpervision and 
Curriculum Development and the Association of American Publishers, and the National 
Committee for Citizens in Education collaborated with the EPIE Institute (Educational 
Products Information Exchange Institute et al., 1969). 

Papers presented at conferences of professional associations constituted twenty 
documents (13.6%) entered in the Resources in Education database. 
Twelve professional associations were represented among this group, with five papers 
being contributed by speakers at conferences of the American Educational Research 
Association (Eisner, 1970; Banks, 1974; Talmage and Walberg, 1977; Hahn, 1978; and 
Welch, 1978), with three papers being contributed by speakers at conferences of the 
National Council of Teachers of English (Washburn, 1978; Zenke, 1981; and Sword, 
1982), and with two papers each being contributed by speakers at conferences of the 
Association for Supervision and Curriculum Development (Garcia and Armstrong, 1978; 
and Komoski, 1980) and the International Reading Association (Olsen, 1968; and 
Whipple, 1968). Each of the remaining professional associations were represented by 
papers presented by a single speaker: the TESOL Conference (Nadler, 1969); the 
NationjU Council for the Social Studies (Fox, 1972); the National Reading Conference 
(Greenlaw et al., 1973); the Southern Conference on Language Teaching (Chapman, 
1975); the Illinois Association of School Librarians (Schmidt, 1975); the National Forum 
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on Excellence in Education (Brunelle et al., 1983); The Council for Exceptional Children 
(De Leon, 1983); and the American Council on the Teaching of Foreign Languages 
(Benevento, 1984). 

3.2.3.4 Federal Education Agencies and Commissions 

Four documents (2.7%) were contributed to the Resources in Education database by 
federal education agencies and commissions. These documents were authored by Carter 
(1971) for the Bureau of Elementary and Secondary Education, Moses and Watt (1976) 
for the Office of Education, Schmidt (1981) in a collaborative effort including the Office 
of the State Superintendent of Public Instruction, and Antell (1981) for the Education 
Conmiission of the States. 

3.2.3.5 Service Centres 

Authors associated with education service centres contributed eight documents (5.4%) 
to the Resources in Education database. These authors were Swisher (1968) and Naegle 
(1970) for the Mid-Atlantic Region Special Education Instructional Materials Center, 
Magisos (1973) for a collaborative effort including the New England Resource Center, 
Blackburn (1974) for the Social Studies Service Center, the Dissemination and 
Assessment Center for Bilingual Education (1977), the Eastern Pennsylvania Regional 
Resources Center for Special Education (1977), Schmidt (1981) for a collaborative effort 
including the Division of Instructional and Professional Services, and Chanihas (1984) 
for a joint effort including the Region I Adult Education Service Center. 

3.2.3.6 State Education Departments and School Districts 

3.2.3.6.1 Adoption Procedures 

The potential for improving the selections of curriculum materials for American schools 
is very likely to occur from the comprehensive use of textbook adoption procedures. 
Textbook adoption procedures have been enacted by all states of the United States. 
Three distinguishable types of procedure have been adopted for textbook selection, 
varying from a centralised modei based- upon a state selection committee to a 
decentralised model within local school districts (Blaunstein, 1983; Duke, 1985; Muther, 
1986). 

In the centralised model, used by twenty states, state selection committees, varying from 
six to thirty members each, use criteria developed at the state level to select textbooks 
for their respective states. The states, applying a centralised model, are Alabama, 
Arkansas, California, Florida, Georgia, Idaho, Indiana, Kentuclq^, Louisiana, Mississippi, 
New Mexico, North Carolina, Oklahoma, Oregon, South Carolina, Tennessee, Texas, 
Utah, Virginia and West Virginia. In the second model, two states - Aiizona and Nevad?: 
- adopt textbooks for grades K to 8 at the state level, whilst school districts select textbooks 
for grades 9 to 12. 

In the decentralised model, school districts in twenty-eight states develop their own 
criteria, which are then used by local selection conmiittees to screen instructional 
materials. The states, applying a decentralised model, are Alaska, Colorado, 
Connecticutt, Delaware, Hawaii, Illinois, Iowa, Kansas, Maine, Maryland, Massachusetts, 
Michigan, Minnesota, Missouri, Montana, Nebraska, New Hampshire, New Jersey, New 
York, North Dakota, Ohio, Pennsylvania, Rhode Island, South Dakota, Vermont, 
Washington, Wisconsin and Wyoming. 
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The extent to which those states that practise some form of the centralised model of 
adoption have formalised its procedures, are of greatest significance for influencing the 
practices of selecting textbooks. After completing an extensive, comparative survey of 
state selection conmiittees in each of the twenty-two states practising the centralised 
model, Duke found considerable variety in their application of procedures for selecting 
curriculum materials. 

Reflecting cross-sections of both educational and lay communities, state selection 
committees recommended textbooks to state boards of education for listing, in all but 
two states. The duration of the adoption process varies from three months to more than 
a year among these states. Curriculum materials proposed for adoption are usually made 
publicly accessible through materials display centres throughout this period- The 
procedures for adoption also vary widely with some states applying objective criteria to 
evaluate curriculum mateuals whilst others make subjective decisions only. Selection 
committees also compile annotations of materials adopted although only nine of these 
states extend annotations beyond basic purchasing information. 
In spite of the extensive application of these procedures, Duke found from a survey of 
textbook administrators in each of the twenty-two states that there are both weaknesses 
and strengths of the centralised textbook adoption procedures in their respective states. 
Tilt controlling of prices of materials over a set time, the assurance of service by 
publishers, and centralised bidding and purchase were the most frequently mentioned 
strengths, whilst the lack of communication among evaluators, the absense of appropriate 
training of evaluators, the short duration for evaluating materials, the long duration of 
the adoption process, the excessive or limited number of materials accepted and the 
inadequate funding of textbook purchases were the most frequently mentioned 
weaknesses. 

Furthermore, Duke concluded from the data gained in his survey that there were seven 
areas for concern in current adoption procedures. 

"1. The criteria used for evaluatio i vary >videly and appear not always to relate clearly enough 

to current instructional practices to be of uuch value; forms and procedures for recording 

evaluations are not always clear. 

2. Appropriate training for evaluators in using specific criteria is clearly lacking. 

3. The apparent duplication of effort at both state and local levels in evaluating instruction- 
al materials raises questions about the efficiency and effectiveness of the process. 

4. Reviews at the state level appear not to have much uniformity and textbook commission 
members themselves frequently do not do the actual revie\\ing but pass that task on to friends, 
colleagues and others; yet these same commission members make the final recommendations 
for adoption. 

5. The translation of evaluators* reviews of text materials into final votes for adoption remams 
unclear in most states. 

6. Time is a clear factor in the adoption process yet no evidence is available to indicate what 
the optimum time might be for the reviewing process or for the length of the adoption cycle. 

7. Economic factors seem to be a major influence in states keeping state-wide adoption prac- 
tices." 

(Duke, C.R. 1985 A Look at Current Staie-wide Text Adoption Procedures, Paper presented 
at the annual meeting of the National Co.udciI of Teachers of English, Spring Cr ''erence, 
Houston,14-15) 

Even before Duke had provided this evidence of the failures of textbook adoption 
procedures, education officials from twenty-two states acted upon their insights into these 
likely failures to form a 'cartel for excellenct' in order to improve adoption procedures 
(Education Week, 1984). Calls were made for textbook publishers to raise their 
standards, for teachers to be involved to a greater extent in textbook selection processes, 
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for teachers to receive training to become effective selection committee members, for 
clearer specification of selection criteria, and for teachers to receive training on how to 
use textbooks in the classroom. 

Muther (1986) has supported these findings and views, but has related their effects more 
closely to practices in states applying the decentralised modeL She indicates that states 
operating decentralised forms of textbook adoption procedures are frequently influenced 
by publishers to purchase textbooks through *best deals' in order to reduce costs. Such 
a practice occurs because few selection committees are required to provide evidence that 
a particular curriculum material is the best for a particular school's educational program. 
In conclusion, the procedures for textbook adoptions, conducted by American 
educational authorities, through either a centralised model or a decentralised model, 
have both weaknesses and strengths. The strengths, and in particular, the weaknesses of 
these procedures are now becoming evident to researchers and educators endeavouring 
to affect desirable changes in the selection of curriculum materials. Whereas reform of 
textbook adoption procedures is beginning to receive attention, it is also recognised that 
American teachers need to be more directly involved in selecting the textbooks they use 
in their classrooms. For instance, the development of a textbook utilisation policy, such 
as the one being promoted by the EPIE Institute, will extend selection from the initiatory 
stage of adoption to an integral process, involving the use of curriculum materials in the 
transactions of classrooms. 

3.2.3.6.2 Publications and Guidelines 

3.2.3.6.2.1 State Education Departments and State Boards of Education 

As a result of the search, eigh 3en sets of guidelines (12.2%) published by state education 
departments and state boards of education, were identified. These guidelines were 
published by the Pennyslvania State Department of Public Instruction (McGarey, 1954; 
and 1967), the Iowa State Department of Public Instraction (1969), the New Mexico State 
Department of Education (1973), the Arizona State Department of Education (1974a; 
1974b), the Indiana State Department of Public Instruction (1974), the Floridi State 
Department of Education (1975), the Georgia State Department of Education (1975), 
the Illinois State Office of Education (1975), the Massachusetts State Department of 
Education (1975), the Indiana State Department of Public Instruction (1977), the 
California State Department of Education (1981), the Illinois State Board of Education 
(1981), the Maryland State Department of Education (1981), the California State 
Department of Education (1982), the Ohio State Department of Education (1983), and 
the California State Department of Education (1984). Since only twelve states are 
represented, these documents are the products of an even more restricted group of state 
education departments. 

An analysis indicates that nine of these documents relate to specifying guidelines of a 
generic type for selecting and evaluating educational products. Most of the documents 
are directed towards school personnel. The remaining eight documents relate to 
specifying guidelines for selecting and evaluating educational products for special groups. 
Two documents relate to specific guidelines for selecting and evaluating social studies 
materials. The remaining six documents are directed to teachers of minority groups for 
the purpose of ident ifying biases in the following areas: male and female roles; ethnic 
and cultural groups; and disabled persons. 

3.2.3.6.2.2 School Districts 
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As a result of this search, only two sets of guidelines (1.4%) published by school districts 
were identified. These guidelines were -authored by Newman (1977) and Darling (1983). 
Additionally, the Alabama University and Mobile County Public Schools (1976) 
collaborated to publish a i t of guidelines. 

3.3 Conclusion 

This descriptive survey shows that the techniques employed in American education for 
adopting, for evaluating, for selecting, and for utilising educational products have 
contributed substantially towards affecting one of the fundamental variables - the uses of 
instructional materials - involved in classroom transactions. 

The precipitating cause that led to the development of these techniques was the rapid 
expansion in production of educational products - textbooks, supplementary print 
materials, audio-visual materials and educational equipment - that occurred during the 
curriculum reform movement of the 1960's. Klein (1978) states that this expansion was 
caused by four factors: tht reassessment of the goals and the achievements of public 
schooling; the dissatisfaction shown by tertiary educators with the educational 
achievements of incoming students; the expansion of knowledge in many disciplines; 
and the growth of civil rights movements among many minority groups. The outcome of 
this movement was the production of a vast quantity of instpactional materials for 
redejSned and new areas of the curriculum. The expansion in the quantity of instructional 
materials, together with the decentralisation of American schooling and the trend 
towards accountability in education, focused attention upon the need for improving the 
quality of this large quantity of materials. 

At the same time, this need was met by the expansion of educational agencies that 
occu. red in the United States during the 1960's. Under the Research and Development 
Centers Program (1963), the United States Office of Education (USOE) established four 
research and development centres during 1964, which were increased to ten centres 
within a few years. The intent of the research and development centres was to identify 
high-priority, educational problems which could either be researched or be treated by 
the synthesis of information available from already completed research. Through 
guidelines provided to the USOE by amendments to the Co-operative Research Act in 
1965, companion regional educational laboratories were founded in 1966 with the 
functions of field-testing and modifying educational procedures and materials developed 
by research and development centres, and then disseminating the resulting innovations. 
It was within this climate that the initial attempts were made to grapple with the problems 
inherent in improving the quality of educational products by relating this issue to the 
curriculum and instructional design. The issue was solved, firstly, by two research efforts: 
that one conducted at the Social Science Education Consortium during the late 1960's to 
develop an instrument for instructional design analysis, the Curriculum Materials Analysis 
SjJtem; and that one conducted by Eash to develop an instmment for instructional design 
analysis at the University of Illinois at Chicago Circle. Secondly, the issues of research 
and development, field-testing, modifying and diffusing procedures on evaluating 
materials and other problems related to educational products, has been accomplished by 
the establishment and the operation of the EPIE Institute. 

The descriptions of programs surveyed in this chapter have shown that the EPIE Institute 
has extended instructional design analysis to establish both internal and external 
congruences of materials and instructional programs. Furthermore, the EPIE Institute 
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has developed several, inventive techniques to improve the quality of educational 
products as well as to relate the use of instructional materials by teachers in classrooms 
to the curriculum. In citing one of a number of other sources that have g mmented upon 
the exemplary practices employed by the EPIE Institute, Scriven (1981: 126) believed 
that the services of the EPIE Institute could be made to yield a thousand-fold upon a 
modest investment, given greater support by state and federal education agencies. This 
has been taken up by the National Commission on Excellence in Education (1983: 28-29), 
which supported the extension of consumer information services of the kind operated by 
the EPIE Institute for purchasers of instruct'^nal materials. 

Whereas issues associated with the evaluation of educational products have been mainly 
solved through the actions of nationally-based agencies, of which the EPIE Institute is 
the most significant, issues related to the adoption and the selection of educational 
products have been addressed mainly by state education departments and school districts. 
The description of adoption procedures has shown that undesirable practices occur, 
although the^e are currently being challenged. 

Furthermore, publishers have been made more accountable for affecting the 
improvement of their products through learner-based verification and revision. The 
need for this improvement through learner-based verification and revision has been 
addressed at a national level, rather than a state or local level. 

This account of the application of techniques to improve the quality and uses of 
educational products in American education does not portray an ideal picture. It does 
provide, however, the potential to be a significant solution to a major issue confronting 
those responsible for the uses of instructional materials in schools. An important, if 
insubstantial, investment in research since the 1960's is now providing a solution which 
relates the uses of instructional materials directly to curriculum development, curriculum 
implementation and curriculum evaluation. The proponents of this solution have also 
effectively bridged the gap between research and practice through the actions of 
educational organisations, such as the EPIE Institute, which combine the function of 
research and development with the provision of services to education departments, 
school districts and other users. 
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CHAPTER 4 



THE PROSPECTIVE PROGRAM 



The purpose of this chapter is to identify a set of objectives and other characteristics upon 
which the prospective program can be specified. These specifications refer to the design 
of a program that applies reliable and valid techniques to collect, to analyse and to 
disseminate qualitative data on instructional materials to the Australian educational 
community on a national basis. 

The context evaluation h?s identified deficiencies in the methods, the techniques and the 
practices applied in the Australian context, both through an examination of the Australian 
context presented in Chapter 2 and through comparison with the American context 
presented in Chapter 3. Recognition of these discrepancies intimates that a prospective 
program would benefit fiom contributions bestowed by foreign educational 
organisations. A subsidiary aim of the prospective program, therefore, is to include scope 
for possible cooperation with three foreign educational organisations; the Educational 
Products Information Exchange (EPIE) Institute; the Social Science Education 
Consoriium (SSEC); and the Canadian Exchange for Instructional Materials Analy.sis 
(CEIMA). The extent of this cooperation will be investigated within the input 
evaluation. 

4.1 The Research Problem 

Baseline data of two types were collected to determine deficiencies: data from documents 
and a multi-site case study collected at a micro-level in Tasmania; and data from state 
education departments collected at a macro-level in other Australian states. 
Deficiencies in the reliability and validity of techniques to evaluate educational products 
have bee n determined through judgments based upon this empirical evidence. 
These deficiencies appear to be a consequence, in part, of shortcomings in the existing 
program. The empirical evidence presented on the existing program for analysing 
educational products in the Australian context shows extensive, but essentially 
uncoordinated, activities occurring at the state level in which techniques of low reliability 
and low validity are generally applied. These activities are not usual'-' supported at the 
national level, with the exception of the current development and implementation of the 
National Software Coordination Unit (NSCU). 
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The research problem is to explicate the objectives and other characteristics of a 
prospective program to be implemented at the national level. Specification of these 
characteristics was facilitated by adapting six categories of criteria exDounded by 
Stufflebeam et al. (1971a): significance of focus; significance of outcomes; uniqueness; 
viability; adequacy of program elements; and cost. 

.2 The Method 

Stufflebeam et al. (1971a), as Advocate Team No. 1, submitted a report to the Division 
of Research and Development Resources, United States Office of Education, which 
proposed an evaluation system for research and development institutions and programs. 
This report includes an organisational fi-amework within which the evaluation system 
could function, procedures to be followed in implementing the evaluation system, and a 
set of criteria and related guidelines for use in applying the proposed evaluative approach. 
A set of criteria was selected and was applied according to the specification established 
by Advocate Team No. 1. Selection of criteria was determined by defining the type of 
classification to which the evaluation fits: prospective institution; prospective program; 
retrospective institution; or retrospective program. It was determined that criteria 
pertaining to the prospective program should be selected. These criteria were then 
matched to guidelines specified by Stufflebeam et al. (1971a) for ihe decision type, 
Program-Plan Approval, one of three types of prospective programs. The guidelines 
were rearranged to conform with the set of criteria extracted, and, in cases where 
guidelines were not specified, the researcher wrote his own guidelines. The guidelines 
for specification of the prospective program are reproduced as Appendix D. 

3 The Specifications of the Prospective Program 

The problem is to describe the objectives and other characteristics of the prospective 
program for collecting, analysing and disi;eminating qualitative data on instructional 
materials (other than computer courseware) to the Australian educational community at 
a national level. These characteristics are described under guidelines specified by 
Stufflebeam et al. (1971a), as follows. 

Sig.nificance of focus 

- Priority 

Guideline 1 

On 22 September 1987, the Minister for Employment, Education and Training presented 
a statement importuning that educational outcomes must be congruent with the 
renuirements of a restructured economy (Dawkins, 1987). A significant role is to be 
given to the higher education system in promoting the federal government's economic 
and social objectives. This statement was followed by a policy discussion paper 
(Australia, Parliament, 1987) intended to elicit responses before new legislation came 
into effect during 1988. This legislation led to a major restructure of national educational 
organisations: firstly, the merging of the Commonwealth Schools Commission with the 
Department of Employment, Education and Training, which includes the Curriculum 
Development Centre as a division; and secondly, the formation of a National Board of 
Employment, Education and Training, to which four advisory councils - the Schools 
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Council, the Higher Education Council, the Employment and Skills Formation Council, 
and the Australian Research Council - report. 

The purpose of establishing this priority for the higher education system is to maximise 
the flexibility of the educational system to respond to national needs, to produce quality 
graduates using available resources, and to remove barriers that impede both change and 
the introduction of innovative approaches. These reforms to the higher education 
system are also having significant implications for determining the priorities of primary 
and secondary education. With a major aim to extend retention rates substantially to the 
end of grade 12, three main implications for primary and secondary education are 
identified in the policy discussion paper: changes to the curriculum, particularly in 
science and mathematics; changes to teacher education and retraining; and changes to 
funding and infrastructure provisions. An emphasis is also placed upon extending 
opportunities to groups identified as disadvantaged: those economically disadvantaged; 
people from rural and isolated areas; and aboriginal Australians. 
On 23 May 1988, the Minister for Employment, Education and Training presented a 
statement inviting cooperation from the states toward a national effort to strengthen the 
capacity of Australian schools (Dawkins, 1988). The statement presented a rationale for 
developing a common curricJum framework for Australian schools. The framework is 
to specify common objectives which can accommodate specific content for regional 
needs, whilst recognising Australia's role in the Asian and Pacific region. A priority is 
to be given to establishing mathematics and science courses that reflect the technological 
needs of Australian society. This curriculum framework is to be supported by a common 
approach to student assessment and reporting. Improving preservice and inservice 
teacher education, increasing the student retention rates in schools, providing for special 
educational needs of disadvantaged groups, and maximising the investment in education 
were recommended in the statement. The statement projected that a process of 
consuhation with the states and school systems would be initiated within two months. 
It can be inferred that the prospective program will be congruent with the educational 
priorities of the government. The intent in the prospective program to design a system 
for the exchange of information on instructional materials will be consistent with the 
educational priorities of the government to enhance higher education and to develop 
national curriculum guidelines for Australian schools. The exchange will serve the needs 
of state education departments, which will be expected to share curriculum resources to 
a far greater extent than previously. 

Guideline 2 

Before its dissolution, the Commonv/eahh Schools Com^Tiission (1987) stated in a recent 
report on education i' at it is crucial five fields of action are addressed through a 
coordinated national effort: the balance, rigour, relevance and cohesion of curriculum 
development; the equity, national compatibility and inclusiveness of accreditation, 
assessment and credentialling; school organisation and climate; integrated on-the-job 
approaches to teacher development; and reciprocal obligations of links with the wider 
community. Guidelines to achieve each of these fields of action are stated for the 
national, state and school levels. 

A key task stated for the national education system is to collaborate with state and 
territorial education departments on the development of curriculum and the exchange 
of information, course design and materials evaluation (Commonwealth Schools 
Commission, 1987: 131). 
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Guideline 3 

The prospective program is predicted to have both social and economic consequences. 
Social consequences are likely to result from gaining a better match between instructional 
materials and student needs. It can be expected that social adjustment of learners toward 
equal opportunities and muhiculturalism will be ameliorated by the identification of 
biases in instructional materials. 

Economic consequences are likely to occur through reducing wasted expenditure on 
educational products, and through providing a degree of consumer protection by means 
of actions on complaints about faulty materials, inadequate service, and unethical sales 
strategies. 

Guideline 4 

All categories of the student population, all demographic areas and all economic settings 
in the Australian educatioiial community will have the potential to be served by the 
prospective program. 

Guideline 5 

It can be expected that the work of the prospective program will continue to be significant, 
even if priorities shift in the future. TTie program is expected to satisfy a continuing need 
for *he provision of qualitative information on educational products. This prediction is 
based upon the continuation of similar programs in foreign contexts of the educational 
setting. 

- Amenability to a research and development approach 

Guideline 6 

A substantial proportion of knowledge and skills pertaining to techniques to provide 
qualitative data on educational products has been gained through research and 
development activities. It can be expected that further advances in knowledge and skills 
are likely to occur through research and development activities. Because research and 
development activities in foreign contexts Lave been responsible for gains in such 
knowledge and skills, a significant and immediate requirement will be the adaptation of 
such activities to suit Australian requirements and conditions. 

Significance of outcomes 

Guideline 7 

The significance of the prospective outcomes lies in the capacity of information on 
instructional materials derived from new and innovative techniques being delivered to a 
substantial proportion of Australian schools. 

Guideline 8 

The prospective outcomes are likely to be both timely and critical. The prospective 
program will introduce innovative methods and techniques to provide information on 
instructional materials at a time when the / stralian edj^ation system is gaining the 
capability to develop information databases and services on a national icale. Because 
of the emphasis toward coordinated national effort, attention is being given, at the same 
time, to rationalisation and elimination of duplicated programs at the state level. 

Uniqueness 
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Guideline 9 

Research and development capabilities in this problem area have not been readily 
established at Australian educational organisations. The creation of the prospective 
program will introduce new research and development capabilities to Australian 
educational organisations. 

Guideline 10 

The distinction of the program lies in its capacity to provide a better match between 
instructional materials and educational programs than hitherto provided. This would be 
accomplished through techniques of qualitative evaluation, and possibly through the 
introduction in the future of quantitative techniques, based upon an integrated 
instructional information resource. 

Guideline11 

Since the techniques and attendant services to be provided by the prospective program 
are not currently available to the Australian educational community, the prospective 
program will play a distinctive role in its geographic area. 

Viability 

- Pidnning capability 

Guideline 12 

The prospective program has been intiated by persons who have a common interest in its 
coming to fruition. Evidence is not available that this group of persons is able to provide 
effective leadership to carry out the program plans. The expertise that these persons 
have gained during the course of the project, however, would place them in prime 
positions as potential leaders and consultants. It is possible that effective leadership 
would only be available from an external source. 

- Potential legal, social, and moral viability 

Guideline 13 

The prospective program appears to satisfy potential requirements for legal, social and 
moral viability in the Australian context. The component in the program, which refers to 
cooperation with foreign educational organisations, is potentially controversial because 
Australian educational organisations have been unaccustomed to incorporating such 
programs in the past. 

The prospective program has the potential address censorship issues 
dispassionately. The use of technique i to measure qualitative and quantitative data on 
biases in instructional materials would provide objective information that educators need 
to counter those who use intuitive arguments to censor the use of controversial materials 
in schools. 

- Cooperability-cooptability 

Guideline 14 

The subsidiary aims of the program are to develop the potential for collaborative 
relationships with other projects and organisations. Establishment of a relationship with 
the NSCU project may lead to standardisation of the processes and the products of both 
programs. Establishment of relationships with organisations from foreign contexts.- the 
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EPIC Institute, the SSEC and the CEIMA - would extend the capabilities for research, 
development, diffusion and adoption within the prospective program. 

Guideline 15 

A pilot study will be conducted as part of this evaluation project to determine what 
support three organisations - the EPIE Institute, the SSEC, and the CEIMA - in North 
America can provide. 

Guideline 16 

The prospective program will apply methods, techniques and practices used by three 
educational organisations in foreign contexts: the EPIE Institute at New York, New York; 
the SSEC at Boulder, Colorado; and the CEIMA, a consortium of three provincial 
education departments in Canada. The prospective program will adapt methods, 
techniques and practices related to work that has been accomplished previously by these 
organisations. Subsequently, state-of-the-art processes and products, such as, an 
integrated instructional information resource will be implemented from current research 
and development work being conducted by the EPIE Institute. 

Parity 

Guideline 17 

It would be expected that users of the ASCIS database would gain parity as users of the 
system. 

Guideline 18 

Representation would b^ extended to non-users of the ASCIS database on the basis of 
the capability of the program to service such users. 

Practicality 

Guideline 19 

".Tie prospective program would embody the properties of accomplishing organisationally 
defined objectives together with exploratory objectives, characterised by a degree of risk 
but promising high payoff. The availability of a system of descriptive analytic and 
evaluative information of records stored on the ASCIS database is consistent with the 
mission of ASCIS to assist teacher-librarians and teachers select instructional materials. 
The application to the Australian context of research undertaken at foreign sources 
involves a risk of its not gaining acceptance but high payoff in providing a better match 
between Instructional materials and educational programs. 

Guideline 20 

Because of the possibility that the attributes of the new methods, techniques and practices 
introduced to Australian education could be misunderstood, an effective plan for 
publicising the prospective program would be needed. Such a plan would direct publicity 
toward current ASCIS users, stressing the innovative techniques used to analyse 
qualitative data on instructional materials. 

Guideline 21 

A starting point for each component of the program is indicated by each objective, which 
is to be integrated at a particular point in the sequence. A milestone point would te 
attained at the end of the implementation of the short-term objectives. The ending point 



set for the prospective program occurs at the conclusion of implementation of the 
long-term objectives. This ending point could be reset in the future so as not to preclude 
the incorporation of objectives, as yet unstated. 

- Personnel 

Guideline 22 

Staff qualifications are relevant to carrying out the tasks of the program. Management 
personnel would need qualifications in relevant areas of educational administration and 
educational evaluation. Professional staff would need qualifications in educational 
evaluation and also in the in-service training progi'am provided by the organisation. 

Guideline 23 

Professional staff, the analysts of instructional materials, would be experienced classroom 
teachers and teacher-librarians employed on a part-time basis. They would be trained 
and certificated in the analytic techniques through the in-service training program. 

Guideline 24 

It is not possible to provide evidence on the commitment of permanent staff members to 
the mission of the program. Selection of staff members has not occurred, as yet. 

Adequacy of program elements 

- Objectives 

Guideline 25 

•The principal aim of the prospective program is to develop, to design and to implement 
a clearinghouse for exchanging qualitative information on instructional materials (other 
than computer courseware) used in Australian schools. Selection of this aim is based 
upon a priority to introduce short-term objectives for an operational system, partly 
research and partly service, and to provide a foundation to implement long-term 
objectives. 

The terminal product will provide new knowledge on curriculum materials. The product 
will be developed by selecting annotated information in the ASCIS database on 
instructional materials of extensive usage in Australian schools. Descriptive analyses 
and evaluations of these instructional materials would constitute the initial parameter of 
the database. Therefore, the aim will establish a balance between providing this novel 
information and supporting available information on the ASCIS database. Achievement 
of this aim is exjpected to make a sharp modification, initially in the knowledge and the 
attitudes of Australian teachers toward using instructional materials and, thereafter, to 
make an incremental improvement in their understandings and skills. 
Listed in order of priority for development and implementation are six short-term 
objectives of the principal aim: 

1. to determine the types of Information to be collected, whether producer, analyst, and 
user information; 

2. to apply a technique of instmctional design analysis for the descriptive analysis and 
evaluation of curriculum materials, that also incorporates criteria to identify biases and 
to match readability levels of curriculum materials and students' reading levels; 

3. to develop a data collection instrument; 
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4. to implement a program to train and to certificate analysts in the adopted technique 
for descriptive analysis and evaluation of curriculum materials; 

5. to develop software for storage and retrieval of information; and 

6. to develop the products for dissemination of information to users. 

Listed in order of priority for implementation are seven long-term objectives of the 
principal aim: 

7. to develop the practical means for publicising the program, and for informing users 
of significant matters concerning educational products, and for collecting information 
from users; 

8. to develop and to implement a teacher education program; 

9. to implement guidelines for Australian publishers of instructional rraterials that 
comply \vith learner-based verification and revision requirements; 

10. to implement standardised techniques and processes for adopting and selecting 
instructional materials through a utilisation policy; * 

11. to initiate research on the uses of instructional materials in the Australian educational 
context; 

12. to investigate the need to provide a consumer protection service; and 

13. to initiate the introduction of an integrated instructional information resource and a 
curriculum alignment service. 

Each objective ^Nould consist of a specified terminal product, as follows. 

1. For objective 1, a pilot study would be initiated to determine the feasibility of 
collecting product producer information, product analyst information, and product user 
information in the Australian context. 

2. For objective 2, the technique of instructional design analysis would incorporate a 
widely accepted model of curriculum development, such as, Tyler's objectives model, an 
ii eraction model, or a process model, a set of criteria to describe biases, and a technique 
tc quantify readability measures in terms of both materials and learners. 

3. For objective 3, the data collection instrument would need to be capable of 
adaptations for the analysis of educational products in different media and to be capable 
of revisions based upon criticisms received from educators. 

4. For objective 4, the training program would consist of a process to ensure that the 
training, the certification and the monitoring of prospective analysts were adequate to 
ensure that an acceptable standard of inter-rater reliability was attained. 

5. For objective 5, the software would be compatible with the requirements of both the 
prospective program and the ASCIS standards, and would be capable of storing and 
retrieving information to fulfil a broadcast mode and an interactive mode. 

6. For objective 6, the products would be available to subscribers through an on-line 
service and, optionally, through print materials. 

7. For objective 7, the practical means would be appropriate to serve the purposes: an 
informative brochure for general circulation to publicise the program; a newsletter to 
inform users of significant issues; and self-report forms to gather data on various matters 
from users. 

8. For objective 8, the guidelines for learner-based verification and revision would 
comply with procedures determined for different techniques (pilot-trial, field-trial, 
pilot-test, and field-test) and with circumstances when each should be administered by 
publishers. 
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9. For objective 9, the teacher development program would emphasise the development 
of modules both as a means of addressing a range of topics on the uses of educational 
products in schools and for disseminating the teacher development program widely. 

10. For objective 10, the utilisation process would consist of three stages: firstly, 
practices to adopt instructional materials would be based upon standardised procedures 
for selection and evaluation; secondly, there would be in-service training on the 
management of instructional materials; and thirdly, there would be in-service training 
on matching curriculum materials to the educational program. 

11. For objective 1 1, a research program would be instigated to analyse quantitative data, 
whether by experimental, quasi-experimental or correlational designs, and qualitative 
data on significant issues related to the uses of instructional materials in the Australian 
educational system. 

12. For objective 12, self-report measurement instruments would be applied in a pilot 
study to determine users' opinions on providing a consumer protection service that would 
act upon complaints. 

13. For objective 13, a pilot study would investigate the means whereby an integrated 
instructional information resource database would be established. 

^^ subsidiary aim is to provide liaison with the National Software Coordination Unit 
(NSCU). The intent of such liaison would be to standardise the processes and the 
products of the two programs, and to maintain the compatibility of other aspects of the 
programs. 

Specific objectives would be determined through consultation with the NSCU. 

subsidiary aim is to consult with the Educational Products Information Exchange 
(EPIE) Institute, the Socml Science Education Consortium (SSEC) and the Canadian 
Exchange for Instructional Materials Analysis (CEIMA) on the nature of cooperation 
between the prospective program and these three organisations. Because the outcomes 
of this aim must be determined beforehand, a pilot study is conducted as part of this 
evaluation project to investigate the implications of such cooperation. The results of this 
pilot study are reported in the input evaluation. 

Specific objectives would be determined following the pilot study and as a result of 
subsequent consultations with the respective organisations. 

- Relevance of objectives to program and institutional .goals 

Guideline 26 

The prospective program will integrate a range of state-of-the-art processes and products 
to collect, to synthesise and to disseminate data on instructional materials. The 
completeness and internal consistency of these components is assumed from similar 
institutional practices in foreign contexts. 

The components and the products of the program would integrate the missions of the 
participating organisations - the ASCIS and the CDC - involved in the prospective 
program as well as to support the development of the NSCU. The elements of research 
and development are related to the mission of the CDC, whilst the elements of diffusion 
and adoption are related to the mission of the ASCIS. The coherence of the prospective 
program would depend largely upon the coordination of these elements. It can be inferred 
that the prospective program will be instrumental for the participating organisations to 
achieve a common mission. At present, the development and the utilisation of 
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curriculum materials form integral components of the work of the CDC, whilst the 
mission of the ASCIS is to compile information on instructional materials. The 
prospective program is likely to integrate such incommensurate components to form a 
mission common to both organise * 

- Adequacy of procedural C5f4>^ 

Guideline 27 

Two types of procedure would be applied to achieve objectives. When an objective would 
require that quantitative data are collected and analysed by means of an empirical 
method, a pilot study would be conducted to achieve such an objective initially. When an 
objective would depend upon knowledge derived from a research base of information, a 
procedure for identifying, retrieving and analysing project reports and other documents 
would be conducted to achieve such an objective initially. A pilot study would be 
designed to implement the objectives operationally, allowing for the successive 
incorporation of each of the objectives. 

These procedures are selected because they are likely to be appropriate to incorporate 
objectives successively over a period of time. Such procedures have been applied 
successfully in foreign contexts. 

MtJ .anisms for both formative and summative evaluation of the prospective program 
would be included in the pilot study. 

- Methodological adequacy 

Guideline 28 

The methodology would be based upon that shown to be appropriate, sufficient and 
economical in foreign contexts - the United States of America and Canada. On this basis, 
methodological adequacy would depend upon support from external sources. 

- Appropriateness of schedule 

Guideline 29 

At the stage of the presentation of this specification, a fixed schedule has not been 
assigned to the prospective program. The schedule and the budgetary allocation would 
be interdependent. The schedule would be planned subsequent to the presentation of a 
budgetary allocation. It is expected that budgetary factors and the program elements can 
be matched to determine an appropriate, sufficient and economical schedule. 

- Adequacy of facilities 

Guideline 30 

The adequacy of the facilities would depend upon the coordination of the present 
facilities of both the CDC and ASCIS. The combination of the facilities of both 
organisations would be appropriate and economical for the work. It would be expected 
that research activities, however, would not be adequately met bv the CDC. In this case, 
research activities would need substantial support from foreign sources. 

Budget 

Guideline 31 

A detailed budget cannot be supplied at the stage of presentation of this specification. 
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It would be expected that the fiscal system would correspond with .hat used by the 
participating organisations. 

It can be expected that organisations from foreign sources, the EPIE Institute, SSECand 
CEIMA, would provide resources to support the planning, effort. 
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CHAPTERS 



THE SYSTEM CAPABILnTES AND 
STRATEGIES 



Deficiencies in the reliability and the validity of the methods, the techniques and the practices 
applied in the Australian context to synthesise data on instructional materials have been 
identified through the contingency evaluation presented in Chapter 2. This judgment is based 
upon comparing die methods, die techniques and die practices applied by educational agencies 
in the Australian and die American contexts. This comparative study led to the specification 
of objective^ dier characteristics for a prospective program, in which reliable and valid 
techniques a. ^ plied to collect, to syndiesise and to disseminate inf -mation on instructional 
materials to die Australian educational community on a national basis. Aldiough derived from 
work conducted in foreign contexts, the specificatioiiji for the prospective program penain to 
current requirements and conditions in Australian education. 

The intent in diis chapter is to present an input evaluation Jiat provides information to meet 
the objectives and odier characteristics of the prospective program. In the first pan. Relevant 
Capabilities of Australian Agencies, the research, die development, the diffusion and die 
adoption activities of two Australian educational agencies, die Curriculum Development 
Centre (CDC) and the Australian Schools Catalogue Information Service (ASCIS), are 
assessed on tVeir capabilities to plan, to structure, to implement and to recycle die procedures 
of the prospective progiam. On die basis cf deficiencies identified through the context 
evaluation, it is determined diat die respective Australian educational agencies must apply 
large change supported by low information grasp. Accordingly, the planned change model is 
applied in such a neomobilistic decision setting. In die second part. Strategies for Achieving 
Program Goals, a survey technique. Policy Implications Analysis (Madey and Stenner, 1981) 
is used to identify die potential input into die prospective program by diree foreign educational 
organisations, die Educational Products Diformation Exchange (EPIE) Institute and the Social 
Science Education Consortium (SSEC) in the Unir^jd States and die Canadian Exchange for 
Instructional Materials Analysis (CEIMA) in Canada. Through generating hypodietical 
findings that may result from planning, structuring, implementing andrecycling the procedures 
of the prospective program. Policy Implications Analysis is applied to elicit data from the 
responding agencies that may determine the contributory role of each to research, 
development, diffusion and adoption activities of the prospective program. In part three, 
Designs for Implementing Strategies, the design for implementing the prospective program is 
determined by establishing die congruence betwt en the findings of ihe two previous studies- 



The design is based upon a combination of the solutions currendy applied by tiie two Austn. '-^Ji 
educational agencies and the adoption of available solutions from foreign contexts. In die 
design for implementing the prospective program, die researcher specifies die need for die 
Australian educational agencies to invite participation from the foreign educational agencies, 
either to initiate or to support planning, structuring, implementing and recycling research, 
development, diffusion and adoption activities of the prospective program. 

5.1 The Relevant Capabilities Of Australian Agencies 

In diis section, the purpose is to describe and to assess die relevant capabilities of die two 
Australian educational organisations, die Curriculum Development Centre (CDC) and die 
Australian Schools Catalogue Information Service (ASCIS), to assume decision-making 
authority and responsibility for die preferential plan of die prospective program oudined in 
Chapter 4. It has been determined dirough die specifications of the prospective program that 
die Curriculum Development Centre (CDC) would assume responsibility for die research and 
development activities, whilst die Australian Schools Catalogue Information Ser\dce (ASCIS) 
would assume responsibility for the diffusion and adoption activities. 

5.1 .1 The Research Problem 

Deficiencies in die mediods, die techniques and die practices applied in Australian education 
to collect, to syndiesise and to diffuse information on instructional materials have been 
identified dirough die collection of baseline data u£ both a imcro-level in Tasmania and at a 
macro-level in odier Australian states. It has also been identified t^--t diese deficiencies are 
amplified by attributes of duplication and incompleteness demonstrated in the existing 
program. 

The research problem is to identify and to assess die relevant capabilities of die two Australian 
educational organisations, the Curriculum Development Centre (CDC) a.id the Australian 
Schools Catalogue Information Service (ACSIS), to assume decision-making audiority and 
responsibility for the preferential plan of die prospective program. 

The study tested die following problem concerning die assessment of die relevant capabilities 
of Au. Uian educational organisations. If substantial deficiencies in die reliability and die 
validity cf techniques were identified dirough data analyses in die context evaluation, dien die 
objectives for large change with a low information grasp are to be specified for a planned 
change model infusing innovative methods, techniques and practices to invent, to test and to 
diffuse solutions to the problem. 

Two procedures were applied to test die research problem: firsdy, die decision-setting and die 
decision model were identified by applying procedures for educational decision-making 
described by Stufflebeam et al. (1971b); and secondly, the relevant capabilities of die two 
Australian educational organisations were a:)Sessed according to the planned change model of 
Clark and Guba(1967). 

5.1.2 The .Method 
5.1.2.1 The Design 

The design applied by die researcher is derived from the procedures for educational 
decision-making described by Stuf^beam et al. (1971b: 49-105). Briefly, these procedures 
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consist of four stages: firstly, defining the problem through the four elements - awareness, 
design, choice and action - of the decision-making process; secondly, determining the 
decision-setting, that is, whether the problem is set in a homeostatic, incremental, 
neomobilistic, or metamorphic decision-setting; thirdly, applying die appropriate decision 
model - whedier it is die synoptic ideal model, die disjointed incremental model, or die planned 
change model - following deteimination of the type of the decision-setting; and fourdily, 
accounting for die types of decisions applied, whedier planning, structuring, implementi..g 
and recycling decisions. 

Once diese procedures had been applied, a flow chart, reproduced as Figure 1, was described 
for planning and strucmring decisions. At the final stage of die flow chart. Programming of 
the Entire Change Process, a procedure for assessing implementing and recycling decisions 
in the activities of research, development, diffusion and adoption for die prospective program 
was identified. This procedure is now described for die planned change model by applying 
eleven processes, each widi specified criteria: research; invention; design; construction; 
assembly; dissemination; demonstration; raining; trial; installation; and 
institutionalisation. The relevant capabilities of the Curriculum Development Centre (CDC) 
are assessed on die criteria diat relate to research and development activities, die relevant 
capabilities of die Australian Schools Catalogue Information Service (AS CIS) are assessed on 
die criteria diat relate to diffusion activities, whilst bodi die Curriculum Development Centre. 
(CDC) and die Australian Schools Catalogue Liformaticn Service (AS CIS) are assessed on 
criteiia for adoption activities. Data are assessed on the objectives and die criteria specified 
by Clark and Guba (1967) for die planned change model. 

5.1.2.2 Data Collection Method 

A specific procedure was not applied to collect data at this stage. Data collected during die 
conduct of the context evaluation formed die basis of information used to determine die 
relevant capabilities of ihe appropriate Australian educational organisations. 

5.1.2.3 Data Analysis 

A procedure, based upon subjective intuitive process, was applied to syndiesise die data and 
to foiTO judgments. 

5.1.3 The Results 

The research problem is to identify and to assess the relevant capabilities of die Curriculum 
Development Centre (CDC) and the Australian Schools Catalogue Information Service 
(ASCIS) to assume decision-making audiority and responsibility for the preferential plan of 
the prospective program. 

Figure 1 illustrates the application of the procedures for educational decision-making, 
described by Stufflebeam et al. (1971b), to die research problem. Tne flow chart, to be read 
from top to bottom, shows diat problem-solving and change fi-om the existing program to die 
prospective program are to be met by application of innovative mediods, techniques and 
practices, derived fi-om foreign contexts, to data on instructional materials. Based upon die 
diagnosis of problems constraining the needs being met, the researcher judged that the 
objectives of die prospective program require significant changes to important variables. On 
the basis of die limited extent of Information available to Australian education on these 
innovative methods, techniques and practices, die researcher determined that a low information 
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FIGURE 2 : A PROCESS CHART DEPICTING THE INITIAL ROLES OF AUSTRALIAN EDUCATIONAL ORGANISATIONS FOR PLANNED CHANGE WITHIN THE PROSPECTIVE PROGRAM 



ACTIVITY 


AGENCY 


PROCESS 
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OBJECTIVE 


CRITERIA 


CAPABILITY 


Research 


Curriculum Development Centre 
(CDC) 


Research 


To advanco knowledge 


Internal and external validity 


Entirely deficient 


Deve lopment 


Curriculum Development Centre 
Centre (CDC) 


Invention 


To formulate a neu solution 
to an ope rat ing problem 


Face validity, estimated 

VO 1 1 U 1 i y t 1 iTipou i 


Mostly deficient 






Design 


T<. draft a plan for 
constructing the program 


Feasibi I ity, tractabi 1 ity 


Mostly sufficient 






Construction 


To build the components 


Design specifications, 
indlviduel ^arformance 


Mostly sufficient 






Assembly 


Vo integrate the components 
Into an operating system 


Design specifications, 

total performance, viability, 

ef f Iclency 


Mostly sufficient 


Diffusion 


Australian Schools Catalogue 
Information Service (ASCIS) 


Dissemination 


To create widespread awareness of the 
program among practitioners 


intelligibility, fidelity, 
pervasiveness, impact 


Mostly sufficient 






Demonstration 


To afford an opportunity to examine 
and assess operating qualities of 
the program 


Credibility, convenience, 
evidential assessment 


Mostly suf> Iclent 


Adopt 1 on 


Curriculum Development Centre 
(COCj and Australian Schools 
Catalogue Information Service 
(ASCIS) 


Tral ning 
Trial 


To train local personnel to manage 
operate, service, utilise the program 

Tf^ hill \ f\ f Ami I t^l+h r\fr\/ye'Atrt 
i\j uu I lu 1 ani 1 1 lai I iy w i i ii iria prOQi ani 

and provide a basis for assessing the 
fit of tht program In a particular 
Institution 


Quantity, continuity, aptitudes, 
motivation, proficiency 

AuopTaD 1 1 iTy , teasiDiiiTy, action 


Mostly deficient 
Mostly sufficient 
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'-jstal latlon 


To fit the characteristics of the 
program to the characteristics of the 
adopting inf.tltutlon 


Effectiveness, efficiency 


Mostly sufficient 
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Institu on- 
al Isatlon 


To assimilate the Invention as an 
integral component of the system 


Continuity, valuation, support 


Mostly sufficient 
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FIGURE 1: A FLOW CHART OF ATTRIBUTES IN DECISION-MAKING 
FOR THE PROSPECTIVE PROGRAM 



NEOMOBILISTIC DECISION SEHING 



Program Operations: to change from the existing progam to 
the prospective program. 





Need and opportunity: a prospective program to apply 
reliable and valid methods, techniques and practices to 
collect, to synthesise and to diffuse data on instructional 
materials applying opportunities derived from foreign contexts. 
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unreliable and invalid techniqi 
the duplication and incompletei 


large change: the use of 
jes in the existing program; 
less of the existing program. 
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Objectives for Large Change: ( 
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Low Information Grasp: Knowledge and skills of innovative 
methods, techniques and practices not extensive. 
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CO 
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Planned change model selected. 
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Programming of the Entire Change Process for Research, 
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Objectives and Criteria to Assess the Process for Research, 
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Institutional isation. 
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grasp prevails. The planned change model was selected on the basis of its prescription by 
Stufflebeam et al. (1971b, 71 -78) for neomobilistic change. 

Figure 2 illustrates the application of the planned change model for implementing and 
recycling decisions to the research problem. On the basis of litde information on the potential 
application of the prospective program to a likely configuration, in which die Curriculum 
Development Centre (CDC) would appropriate the main responsibilities for the research and 
development activities whilst the Australian Schools Catalogue Information Service (ASCIS) 
would appropriate the main responsibility for the diffusion activity and co-operate in sharing 
responsibility with the Curriculum Development Centre (CDC) for die adoption activity, die 
researcher judged die capabilities of die respective agencies on each objective of the change 
process, as follows. The researcher applied a Likert-type rating scale of five points, labelled, 
'entirely sufficient', *mosdy sufficient', 'uncertain', 'mosdy deficient' and 'entirely deficient', 
to assess potential capability. 

1. For the objective on research, the capability of die CDC to advance knowledge diat provides 
a basis for invention is judged to be entirely deficient. 

2. For die objective on invention, die capability of die CDC to formulate new solutions to 
operating problems diat produce inventions is judged to be mosdy deficient. 

3. For the objective on design, die capability of die CDC to draft a blue-print that fits the targe 
situation is judged to be mostiy sufficient. 

4. For die objective on construction, die capability of the CDC to construct the components 
for implementing die design is judged to be mosdy sufficient 

5. For die objective on assembly, die capability of die CDC to integrate the components to 
produce a co-ordinated operating system is judged to be mosdy sufficient 

6. For die objective on dissemination, die capability of die ASCIS to create widespread 
vareness of die prospective program is judged to he mosdy sufficient. 

'/ For die objective on demonstration, die capabl.ity of die ASCIS to afford opportunities to 
examine and to assess the operating qualities of the prospective program is judged to be mosdy 
sufficient 

8. For die objective on training, die capability of bodi the CDC and die ASCIS to train personnel 
to manage, to operate, to service and to utilise the prospective program is judged to be mosdy 
deficient. 

9. For die objective on trial, the capability of bodi die CDC and the ASCIS to trial die 
prospective program in the context of a particular situation is judged to be mostiy sufficient. 

10. For die objective on installation, the capability of die CDC and the ASCJS to operationalise 
the prospective program widiin the adopting agencies, the CDC and the ASCIS, is judged to 
be uiosdy sufficient 

11. For the objective on institutionalisation, die Capability of the CDC and die ASCIS to 
establish the prospective program as part of an ongoing program is judged to be mosdy 
sufficient 

5.1.4 The Discussion 

The study identified and assessed the relevant capabilities of the two Australian educational 
organisations, die Curriculum Development Centre (CDC) and die* Australian Schools 
Catalogue Informarion Service (ASCIS), to assume decisioa-making authority and 
responsibility for die preferential plan of the prospective program. 

It was assumed that these agencies would be deficient in implementing and recycling decisions 
when valid and reliable techniques to collect, to syndiesise and to diffuse information on 
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instructional materials are to be applied extensively to tt^Jse activities, because of the 
substantial deficiencies identified in the sorts of techniques applied in current Australian 
practice. On the other hand, it was assumed that the agencies would be sufficient in 
implementing and recycling decisions when these techniques are not to be applied extensively 
to such activities. 

The results of the application of the planned change model to the research problem showed 
that the Australian agencies assuming responsibility for the preferential plan of the prospective 
program vary in tiieir capabilities for implementing and recycling decisions, as follows: tiiey 
are deficient in research activities; they are sufficient in most development activities; they 
are sufficient in diffusion activities; and tiiey are sufficient in most adoption activities. 
The results indicate tiiat large change is required to structure, to implement and to recycle 
decisions in cases when techniques to collect, to synthesise and to dif formation on 
instnictional materials are to be applied extensively. Based upon a lov. if information 
grasp, such an assess^.ient raises rhe necessity that the decision uk o support the 
prospective program requires the projection of an extensive set of steps to accomplish such 
lar^^e change. This is needed if tiie prospect for success of the prospective program is to be 
met. 

This conclusion does not only apply to tiiose activities in which techniques to collect, to 
synthesise and to diffuse information on instructional materials are to be applied extensively, 
but also is addressed to those activities in which these techniques are not applied extensively. 
In no case was tiie capability of tlie Australian agencies judged to be entirely sufficient. The 
potential effectiveness of the Australian educational agencies to conduct these activities cannot 
be assumed. Neither evidence to this effect is available nor have comparative studies been 
conducted to assess the achievements of American and Australian educational agencies to plan, 
to structure, to implement and to recycle decisions. In spite of tiiis lack of information, it is 
reasonable to presume that Australian educational agencies, which are still working at a 
developmental stage in accomplishing tiie capabilities to plan, to structure, to implement and 
to recycle decisions, are less able to perform tiiese processes than more established American 
educational organisations. 

Strategies Fcr Achieving Program Goals 

In this section, the purpose is to assess the strategies that foreign educational organisations 
might employ to achieve the goals of the prospective program. This assessment was made 
through the administration of a questionnaire, Survy of Attitudes on Potential Foreign Input 
into the Prospective Program in Australian Education, to tiie staff members of three foreign 
educational organisations: tiie Educational Products Information Exchange (EPIE) Institute; 
the Social Science Education Consortium (SSEC); and the Canadian Exchange for 
Instructional Materials Analysis (CEIMA). The researcher had identified that each of tiiese 
three educational organisations maintained the potential to extend some form of involvement 
into the prospective program for Australian education defined in tiiis report. In each case, tiie 
researcher had identified such potential through communications during an extensive period 
in which the staff members of each organisation had expressed either explicidy or implicidy 
an interest in collaborating witii Australian educational organisations on work in tiie area of 
the prospective program. 

lA The Research Problem 
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The context evaluation of this research project has indicated that the relevant capabilities of 
the Curriculum Development Centre and the Australian Schools Catalogue Information 
Service to provide research, development, diffusion and adoption activities of the prospective 
program are adequate in some attributes but inadequate in other attributes. It has been 
identified that significant inadequacies exist in institutional capability, program elements, 
management, personnel, funding, schedule, facilities and comiiiunication for the research 
activity, for the development activity of invention and for the adoption activity of training. 
The study sought to determine the following general objective concerning strategies to achieve 
the goals of the prospective program. The general objective is to determine the extent of 
potential foreign input into the prospective program in Australian education. 
If the relevant capabilities of the Australian educational organisations to be coopted into the 
prospective program are inadequate, in part, to implement the prospective program 
successful!), a case c^n be made that educational organisations conducting similar programs 
successfully in foreign settings should be provided with the opportunities to contribute new 
methods, techniques and practices to support the prospective program. The study investigated 
three specific objectives concerning the attributes of the potential foreign input into the 
prospective program in Australian education: firstly, the relationship between forecasted 
scenarios for the prospective program and the capability of the foreign educational 
organisations to provide input need to be identified; secondly, the attributes of policy actions 
on the part of foreign educational organisations, arising from the forecasted scenarios, need 
to be identified; and thirdly, recommendations on the part of foreign educational organisations 
to revamp the prospective program need to be elicited. 

5.2.2 The Method 

5.2.2.1 The Target Population 

The target population for the Survey of Attitudes on Potential Foreign Input into the 
Prospective Program in Australian Education comprised a panel of staff members of three 
foreign educational organisafons, which have as their missions, the collection, the synthesis 
and the diffusion of data on instructional materials. Selection of these organisations had been 
based initially upon a perception that each of these organisations possessed the potential to 
contribute to Australian education innovative methods, techniques and practices on the uses 
of instructional materials. Prior to the conduct of tlie survey, a response on the prospective 
participation of each organisation was sought through correspondence. 
The staff members, who responded to the survey, belonged to the three foreign educational 
organisations listed below. 

Site 1: iTie Educational Products Information Exchange (EPIE) Institute located at the site of 
the Teachers College, Columbia University, New York, New York, United S tates of America. 
Site 2: The Social Science Education Consortium (SSEC), located at the site of the University 
of Colorado, Boulder, Colorado, United States of America. 

Site 3: The Canadian Exchange for Instructional Materials Analysis (CEMA), a consortium 
of the Ministry of Education, British Columbia, Alberta Education, and Manitoba Education, 
Canada. 

5.2.2.2 The Measurement Instrument 

The questionnaire. Survey of Attitudes on Potential Foreign Input into the Prosp^^ctive Program 
in Australian Education, is based upon an example of the type of instrumentation applied to 
Policy Implications Analysis reported by Madey and Stenner (1981:32-36). This type of 
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measurement instrument has been used by the NTS Research Corporation, Durham, North 
Carolina, in several longitudinal evaluations of federal educational programs in the United 
States, including the State Capacity Building Program for the National Institute of Education 
(NIE), United States Department of Education. This questionnaii-e is derived from the 
measurement instrument administered as part of that example of Policy Implications Analysis. 
The development of the questionnaire followed the procedure used by Madey and S tenner. 
Eight of the twelve major components of criteria, specified in the guidelines for proposals 
(Stufflebeam et al., 1971a: 102-183), were modified to form the parts of the questionnaire: 
institutional capability; program elements; management; personnel; funding; schedule; 
facilities; and communication. 

Development of this instrument passed through two stages. To begin, generation of a 
hypothetical finding for each of the criteria represented the first stage. Each statement 
contained a hypothetical, but theoretically possible, finding which could result from the 
implementation of the input evaluation of this project Each Finding Statement, the item of 
the questionnaire, was intended to represent the likely outcome of its implementation. A set 
of Guidelines, Significant Policy Implications and a Funding Statement, based upon tiiose 
applied by Madey and Stenner, constituted the response section of each part. 
The second stage in the development of the questionnaire concerned its revision. It was 
recogni.':ed that either of two likely scenarios could be represented by the implementation of 
the input evaluation. This caused the researcher to develop hypothetical findings that 
represented each scenario. At the same time, it was recognised that die )pe of the Program 
Elements, derived from the thirteen objectives stated in Chapter 4, was o extensive. It was 
decided dien diat die hypodietical findLng for the Program Elements .vould be restricted to die 
six, short-term objectives. At diis point, die researcher felt diat bodi die lengdi of die 
questionnaire and time required to complete it were too extensive. It was decided diat a balance 
could be obtained between diese elements and die quantity of data by combining pairs of diose 
other components, exhibiting common relationships, to form single finding hypodieses, as 
follows: manag'^-nent and pjrsonnel; funding and schedule; and facilities and 
coirjnunication. The final form of die questionnaire, reproduced as (Appendix E, was 
composed through these revisions. 

.2.2.3 Design 

The design of die procedures for die study is determined by die requirements of Policy 
Implications Analysis defined by Madey and Stenner. Policy Implications Analysis is a 
procedure designed to enhance the likelihood diat an evaluation will have an impact upon 
decision-making. The Policy Implications Analysis method applies two futures techniques: 
die Delphi Method; and Scenario Writing. The method employs six steps: firsdy, die 
generation of hypothetical findings; secondly, die preparation of a questionnaire to be 
administered to a selected panel of respondents; diirdly, the selection of a panel of respondents; 
founhly, the administration of die questionnaire to the panel; fifdily, die analysis of die 
responses; and lasdy, the use of die analysed responses to develop a set of policy-relevant 
hypotheses. 

It is apparent that Policy Implications Analysis is a useful design to apply to input evaluation. 
Ostensibly, Policy Implications Analysis is applied for this purpose in reported studies (Madey 
et al., 1980; McNeil et al., 1980). The essential difference between its application in die 
reported studies and this study pertains to die mode of context to which Policy Implications 
Analysis is applied. In die reported studies. Policy Implications Analysis has been applied 
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within congruence evaluations. In this study, the intention is to apply Policy Implications 
Analysis in the contingency mode. 

5.2.2.4 Data Collection Methods 

Acopy of die questionnaire, Survey of Attitudes on Potential Foreign Input into die Piospective 
Program in Australian Education, was circulated by mail to a designated staff member of each 
educational organisation during April, 1988. A request was made tiiat die completed 
questionnaire should be returned at the close of June, 1988. This deadline was tiien extended 
until the close of September, 1988 for two reasons: first, die. staff members of two educational 
organisations, the Social Science Education Consortium and the Canadian Exchange for 
Instructional Materials Analysis, replied expressing difficulties in completing the 
questionnaire; and second, another staff member of the Canadian Exchange for Instructional 
Materials Analysis, working in a different Canadian province, expressed an interest in 
responding to the questionnaire. 

5.2.2.5 Data Analysis 

A procedure, based upon subjective intuitive process, was applied to syndiesise the data and 
to form judgments on the findings of die Survey of Attitudes on Potential Foreign Input into 
the Prospective Program in Australian Education. 

5.2.3 The Results 

The research problem is to identify and assess die strategies diat diree foreign educational 
organisations can apply to achieve die goals of the prospective program for Australian 
education. 

5.2.3.1 The Educational Products Information Exchange (EPIE) Institute 

Komoski (1988) stated that die EPIE Institute intended to respond to die Survey of Attitudes 
on Potential Foreign Input into the Prospective Program in Australian Education by die 
deadline atthe close of September 1988. Receipt of the completed questionnaire administered 
during the survey, however, had not been received at the timo of vmting this statement for die 
report. It is anticipated that the results of die Survey wiU be presented as an Addendum to 
this report at a future date. 

Komoski (1988) did present a general statement confirming uic continued interest of the EPIE 
Institute in the proposal being developed in this report, as follows. 

"EPIE continues to have a great interest in woridng out a means of assisting Australian educators 
to improve the evaluation, selection, and use of instructional materials." 

5.2.3.2 The Canadian Exchange for Instructional Materials Analysis 
(CEIMA) 

Representing Manitoba Education within the consortium of three provincial education 
departments constituting the CEIMA, Parasuik (1988) completed the Survey of Attitudes on 
Potential Foreign Input into the Prospective Program in Australian Education, and also 
presented the following summary statement. 

"In general, I can state that we would be willing to share information on our experiences in es- 
tablishing our Materials Selection Process by phone or through the mail, in response to specific 
requests. We are interested in pursuing the possibility of more involved activities in the form of- 
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TABLE 7. 

RESPONSES TO GUIDELINES FOR HYPOTHETICAL FINDINGS 
CEIMA (MANITOBA EDUCATION) 

Guideline Institutional Institutional Program Management Funding Facilities 

Capability Capability Elements Personnel Schedule Comm- 

(Preferential) (Optional) (Objectives unicatior 

1-6) 



1. To what degree is this 
finding within the purview 
of your organisation? 



I. How much knowledge do 
you have of the general 
area addressed by the 
findings? 



3. To what degree does this 
finding correspond with 
your expectations? 



To what degree does this 
finding have immediate 
policy implications for a 
potential relationship be- 
tween the prospective 
program and your or- 
ganisation? 



5. Given that this finding 
reflects reality, is it stated 
in a concise and clear 
fashion? 



Note 1. Key to interval scale 
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positive negative 
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providing training in the EPDE or CEIMA process. Such potcntiiil involvement, of course, would 
be dependent on the cost implicadons to Manitoba Educadon." 
Table 7 shows the reponses to each guideline provided by Manitoba Education. The results 
have clear implications for significant input by Manitoba Education into the pi eferred scenario 
for the prospective program. The results indicate that Maniioba Education may be prepared 
to provide important input into the subactivity of training within the prospective program. 
Tht respondent generated significant policy actions for hypothetical findings on institutional 
capability (preferential) and institutional capability (optional). For \h\^ preferential form of 
institutional capability, the repondent generated the following policy action. 

"None, provided assistance is limited to sharing data and informadon on Learning Materials pro- 
cedures." 

For the optional form of institutional capability, the respondent generated the following policy 
action. 

"A decision to commit fiscal resources to an off-shore project" 
The respondent expressed the need for further information in order for policy actions to be 
generated for hypothetical findings on the program elemeiits (objectives 1 to 6), 
manage, ment-personnel, and funding-schedule. For objectives 1 to 6 of the program elements, 
the resplendent expressed the need for: 

"Informadon as to whether or not costs associated v;ith u^ning-inservice, etc., would be covered 
by Australian agencies." 

For the management-personnel component, the respondent expressed the need for: 
"Clarificadon regarding proposed uainbg, consultauve modes, cost implicadons, etc." 

For the funding-schedule component, the respondent expressed the need for: 

"Detailed project descriptors, possible ume commitments, possible communicadon Unks, and 
costs." 

No responses to Finding Statements were presented by this respondent. 
Although staff members representing the Ministry of Education, Provijice of British Columbia 
within the consomum constituting the CEIMA failed to respond to tiie questionnaire, tiie 
following statement, received from McArthur (1988), presents implications for establishing 
future contact between the personnel of the prospective program and the Learning Resources 
Branch of tiie Ministry of Education, Province of British Columbia, Canada. 

"Since our initial correspondence last year, the Ministry of Educadon in Bridsh Columbia has 
reorganized snme of its Branches. The acquisidon and selection of learning resources has been 
made the responsibility of a new Branch endded Learning Resources. 
This Branch has as part of its mandate die bringing togedier of die various evaluadon formats 
currendy in use for print, video and software widi die intent to evaluate all resources from a com- 
mon perspccdve. 

For diese reasons, I cannot proceed v/idi definidve informadon on die role of CEIMA in terms 
of your proposal. As I mendoned in my last letter, your quesdonnaire was extremely complex and 
quite site and context specific. It would be difficult to determine your organizadonal needs, given 
diat oiir own policies and procedures may undergo revision." 

5.2.3.3 The Social Science Education Consortium (SSEC) 

Although the staff members of the Social Science Education Consortium failed to respond to 
the questionnaire, the fc^ wing statement received from Singleton (1988) presents an 
implication for the results of the survey. 

"I received your recent letter regarding our problems in responding to your quesdonnaire and 
have discussed die issue wid\ our director and odier staff members. We gave serious considera- 
don to your proposed 'compromise' forms of responding; however, because we are a specialised 
organisadon focusing on social studied/social science educadon and your project will involve all 
curricular areas, we do not feel well qualified to participate ..i L^e project Certainly, an organic - 
don widi a broader curriculum base, like EPEE, coulu more readily provide die services for which 



you are looking. Thus, we feel that we cannot invest time in responding to the questionnaire at 
this time." 

5.2.4 The Discussion 

The research problem is to identify and assess three foreign educational organisations on the 
strategies that each can apply to achieve the goals of the prospective program for Australian 
education. 

The major findings of the study reported are conclusive with regard to the strategies that the 
Canadian Exchange for Instructional Materials Analysis (CEIMA) and the Social Science 
Education Consortium (SSEQ can apply, but inconclusive with regard to the strategies that 
the Educational Products Information Exchange (EPIE) Institute can apply to the prospective 
program. Within the consortium of the CEIMA, Manitoba Education indicated that it will be 
prepared to share with Australian educators information on establishing its system for 
collecting, synthesising and disseminating information on instructional materials and, perhaps, 
to contribute towards the subactivity of training. On the other hand, conclusive information 
concerning the potential involvement of the Ministry of Education, Province of British 
Columbia was not forthcoming. The Social Science Education Consortium (SSEQ indicated 
conclusively that it did not maintain the resources to contribute stategies to support the goals 
of the prospective program. Although the EPIE Institute did not respond to the survey within 
the specified time, this organisation is presumed to be the foreign organisation that can make 
the most extensive contribution of strategies to support the goals of the prospective program. 
It is predicted that this support could extend to the EPIE Institute providing strategies to achieve 
the goals of the prospective program for each of the eleven subactivities: research; invention; 
design; construction; assembly; dissemination; demonstration; training; trial; installation; and 
institutionalisation. The actual extent of this potential support needs to elicited and identified 
through further communication with the EPIE Institute and research on the topic. 
The main limitation of this study is the incompleteness of the reported data. Consequently, 
it is possible only to present a tentative plan of the procedural design for the prospective 
program on the basis of available data. A second limitation of the study is predicted to be a 
possibility that the findings are biassed because of the likelihood that the respondents would 
be unfamiliar with organisations and programs in Australian education. 

5.3 Designs for Implementing Strategies 

In this section, the purpose is to determine the congruence between the findings of the two 
previous studies. The optimal procedural design for implementing and recycling activities in 
the prospective program is most likely to be determined by establishing the congruence 
between the capabilities of the Australian organisations and the strategies that the foreign 
organisations can employ to achieve the goals of the prospective program. 

5.3.1 The Research Problem 

In the first study presented in this chapter, the relevant capabilities of the Australian educational 
organisations, the Curriculum Development Centre (CDC) and the Australian Schools 
Catalogue Information Service (ASCIS), have been assessed as entirely deficient in the 
subactivity of research, mostly deficient in the subactivities of invention and training, but 
mostly sufficient in the subactivities of design, construction, assembly, dissemination, 
demonstration, trial, installation and institutionalisation. From among the three foreign 
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educational organisations surveyed in the second study, both the Educational Products 
Information Exchange (EPIE) Institute and the Canadian Exchange for Instructional Materials 
Analysis (CEIMA) possess the latent capabilities to employ strategies to achieve the goals of 
the prospective program. The capabilities of this potential have been assessed in the second 
study for the CEIMA but not for the EPIE Institute. 

The research problem is to establish the congruence between the relevant capabilities of the 
Australian educational organisations and the strategies that foreign educational organisations 
can employ to achieve the goals of the prospective program. 

The study tested the following question concerning the congruence between the relevant 
capabilities of the Australian organisations and the strategies to be employed by the foreign 
educational organisations. If deficiencies exist in the capabilities of Australian organisations 
to implement and recycle the research, development, diffusion and adoption activities of the 
prospective program, then these deficiencies can be ameliorated most effectively through the 
application of strategies by foreign educational organisations to achieve the program goals. 
The research plan entailed testing the problem at two stages: at the first, the means whereby 
the Australian educational organisations would structure decisions on the optimal procedural 
design for the prospective program were planned; and at the second, the means and ends 
whereby both the Australian and foreign educational organisations would implement and 
recycle the optimal procedural design for the prospective program were assessed. 

5.3.2 The Method 



5.3.2.1 The Design 

The design applied by the researcher extends the application in tiie first study of the procedures 
for educational decision-making described by Stufflebeam et al. (1971b: 49-105). In this 
study, the procedures for decision-making are applied to produce an end product of the input 
evaluation, an analysis of the procedural design in terms of the collaborative relations between 
both the Australian and foreign educational organisations. The planned change model is 
applied to test the problem at each of two stages: at the first, the model is used to provide an 
action plan whereby the Australian educational organisations would structure the processes of 
the prospective program; and at the second, the model is used to present the optimal procedural 
design whereby both the Australian and foreign educational organisations would implement 
and recycle the processes of the prospective program. 

At the first stage, the taxonomy used by Stufflebeam et al. (1971b: 71-78) to describe the 
change process for structuring decisions in the activities of research, development, diffusion 
and adoption is arrayed against the roles of agencies, the objectives, some criteria associated 
with the objectives, and the relationship of the objectives to the change process in Figure 3. 
The use of this taxonomy allowed the researcher to structure the decisions of the change process 
so that the congruence between the relevant capabilities of the Australian educational 
organisations and the strategies that the foreign educational organisations might employ 
together to implement and recycle decisions in the change process could be assessed at the 
second stage. In this way, it is possible to describe and classify the attributes for structuring 
decisions of the change process in the prospective program. 

At the second stage, the taxonomy used by Stufflebeam ei al. (1971b: 71-78) to describe the 
change process for implp.menting and recycling decisions in the activities of research, 
development, diffusion and adoption, is arrayed in Figure 4 against seven of twelve guidelines 
or categories specified by Stufflebeam et al. (1971a): program elements; management; 
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personnel; funding; schedule; facilities; and communication. The attributes assessed in the 
study were judged on each of three criteria derived from Stufflebeam et al. (1971b: 293-302): 
administrative role; interface role; and technical role. These criteria, defined by Stufflebeam 
et al. (1971b) as encompassing the essential components of evaluation specialists' roles, were 
selected as appropriate to describe the roles of educational organisations in the change process 
within the prospective program. Administrative roles entail directorship, management and 
coordination, interface roles entail identifying decision situations and provision of relevant 
information for decision-making, and technical roles entail measurement, data collection and 
processing, design, and statistical analysis. The use of these procedures in the research design 
allowed the researcher to make global assessments on each criterion by classifying and 
controlling the attributes examined at the second stage. 

Funhermore, the assessment made of the problem, as tested at the second stage of this study, 
is conditional upon decision-makers taking account of two sets of problems on nesting referred 
to by Stufflebeam et al. (1971b: 87-93): firstly, problems arising from multiple 
decision-makers relating to the same decision; and secondly, problems arising from the 
interdependency of decisions. For the first set of problems, Stufflebeam et al. (1971b) caution 
evaluators to recognise potential conflict situations arising from these problems without 
recommending action. They state that educational evaluators lack the capabilities to meet the 
evaluation needs posed by these types of problems. Stufflebeam et al. (1971b) also caution 
evaluators to recognise two types of problems arising from the interdependency of decisions: 
firstly, previous decisions have a bearing on future decisions; and secondly, higher order 
decisions influence or constrain lower order decisions. 

5.3.2.2 Data Collection Method 

A specific procedure was not applied to collect data in this study. For data applicable to the 
Australian context, those applied in this study had been collected during the context evaluation 
and processed in the first study. For data applicable to foreign contexts, those collected during 
the conduct of the second study formed the basis of information in this study. 

5.3.2.3 Data Analysis 

A procedure, based upon subjective intuitive process, was applied to synthesise the data and 
to form judgments during the second stage. By analysing the data obtained during the first 
study, an assessment was made of the e\cent to which the Australian educational organisations 
could achieve the goals of the prospective program on specific subactivities. This analysis 
was matched against an assessment of the extent to which the foreign educational organisations 
could support the goals of the prospective program on specific subactivities from the results 
derived in the second study. From these analyses, global judgments were made on the types 
of relations between the different educational organisations, both Australian and foreign, 
needed to implement and recycle decisions in the optimal procedural design for the prospective 
program on categorical levels for the three criteria: administrative role; interface role; and 
technical role. 

5.3.3 The Results 

The research problem is to establish the congruence between the relevant capabilities of the 
Australian educational organisations and the strategies that foreign educational organisations 
can employ to achieve the goals of the prospective program. 
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FIGURE 3 : AN ACTION PUN DEPICTING THE PROCESS FOR STRUCTURING DECISIONS IN THE 
OPTIMAL PROCEDURAL DESIGN FOR THE PROSPECTIVE PROGRAM 
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FIGURE 4 : AN INTEGRATED PLAN DEPICTING COLLABORATIVE RELATIONSHIPS BETWEEN EDUCATIONAL ORGANIZATIONS FOR 

IMPLEMENTING AND RECYCLING DECISIONS IN THE PROSPECTIVE PROGRAM 
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tATIVE RELATIONSHIPS BETWEEN ORGANISATIONS 

CDC performs administrative, Intei face and technical roles 

CDC performs administrative and Interface roles but collaborates with EPIE on the technical role 
CDC performs the administrative role bjit collaborates with EPIE on the Interface and technical roles 
CDC and EPIE collaborate on administrative. Interface and technical roles 
ASCIS performs administrative, Int^^rface and technical roles 

ASCIS performs administrative and Interface roles but collaborates with EPIE on the technical role 
ASCIS and EPIE perform administrative. Interface and technical roles 
CDC and ASCIS perform administrative, Interface and technical roles 

CDC and ASCIS perform administrative and Interface roles but collaborate with EPIE on the technical role 
CDC and ASCIS perrorm the administrative role but collaborate with EPIE on the interface and technical roles 
CDC, ASCIS and EPIE collaborate on administrative. Interface and technical roles 

CDC and ASCIS perform the administrative and Interface roles but collaborate with EPIE and CEIMA on the technical role 
CDC and ASCIS perform the administrative role but collaborate with EPIE and CEIMA on the Interface and technical roles 
CDC, ASCIS EPIE and CEIMA collaborate on the administrative, Interface and technical roles 




Figure 3 illustrates the application of the planned change model for structuring decisions in 
the prospective program. An action plan for the prospective program is presented whereby 
the Curriculum Development Centre (CDC) will be able to structure decisions for its optimal 
procedural design. The plan illustrates a process whereby the CDC reviews existent research 
on the problem area, implements a set of recommendations and collaborates with the EPLE 
Institute to design, construct and assemble the plan for the prospective program. Then, the 
Australian Schools Catalogue Information Service (ASCIS), collaborating with the EPIE 
Institute, disseminates and demonstrates the plan for the prospective program to client groups 
and key audiences, and both the CDC and the ASCIS, together with the EPIE Institute and the 
CEIMA, train local personnel, trial, install, and institutionalise the prospective program in the 
Australian context. 

Figure 4 illustrates the application of the planned change model for implementing and 
recycling decisions in the prospective program. The process chart depicting the optimal 
procedural design for the prospective program shows that both Australian and foreign 
educational organisations collaborate within the categories of the prospective program. The 
chart illustrates an array of collaborative relationships between the CDC, the ASCIS, the EPIE 
Institute and the CEIMA based upon assessments for each of the seven categories, matched 
against each of the subactivities within the process for implementing and recycling decisions 
for the optimal procedural design. Only for the category. Program Elements, is extensive 
collaborative relationships between the CDC, the ASCIS, the EPIE Institute, and also the 
CEIMA for the subactivity of training, assessed as presenting the optimal procedural design 
for implementing and recycling decisions in the prospective program. The category. Funding, 
is also assessed to require the establishment of collaborative relationships between the CDC 
and the EPIE Institute for the subactivities of research and invention, and between the 
Australian organisations, the EPIE Institute and the CEIMA for the subactivity of training. 
The formation of collaborative relationships between the Australian and foreign educational 
organisations is judged to be unnecessary to implement and recycle decisions for the remaining 
categories in the prospective program. In summary, it is judged that the optimal procedural 
design for implementing and recycling decisions in the prospective program requires that 
collaborative relationships be established between Australian and foreign educational 
organisations in a variety of configurations for the subactivities of research, invention and 
training. 

5.3.4 The Discussion 

The research problem is to establish the congruence between the relevant capabilities of the 
Australian educational organisations and the strategies that foreign educational organisations 
can employ to achieve the goals of the prospective program. This congruence is established 
within the process for implementing and recycling decisions in the prospective program by 
using the planned change model. 

The findings of this study indicate that the Australian educational organisations would need 
to collaborate with the specified foreign educational organisations on all subactivities of the 
category. Program Elements, as well as aU the remaining categories of the subactivities of 
research, invention and training to implement and recycle decisions in the optimal procedural 
design for the prospective program successfully. 

One could postulate that either of two alternative procedural designs should have been 
accepted. Representing dichotomous types, it could be specified in the first case that the 
Australian educational organisations should implement and recycle the procedural design for 




the prospective program without recourse to establishing cooperative relationships with 
foreign educational organisations, whilst in the second case it could be specified that the 
Australian educational organisations should establish collaborative relationships with foreign 
educational organisations to implement and recycle activities for each category of the 
prospective program. Both of these alternative types of procedural design are rejected. The 
first is rejected because it was identified through the context evaluation that the responsible 
Australian educational organisations do not maintain the capabilities for research, invention 
and training to implement and recycle the activities of the prospective program successfully. 
The second is rejected for several practical reasons: the foreign educational organisations do 
not seem to possess the capabilities to ext<='ndcollaboative relationships to the extent necessary 
to facilitate this type; the financial costs of this type are more likely to be prohibitive; and the 
benefits of more extensive collaborative relationships presented by this type are difficult to 
assess. 

5.4 Conclusion 

The input evaluation presented in this chapter has been accomplished in three stages: firstiy 
the relevant capabilities of the two Australian agencies, which are likely to adopt the 
prospective program, have been assessed through use of the planned change model; secondly, 
the strategies which three foreign educational organisations might employ to achieve the goals 
of the prospective program are assessed through the use of policy implications analysis; and 
thirdly, a design for implementing the strategies of the prospective program has been 
determined through establishing a congruence between the findings of the previous two stages. 
Stufflebeam et al. (1971b: 87) have drawn attention to the need for evaluators, who apply the 
planned change model in neomobilistic settings, to obtain and apply an extensive amount of 
informatiori about the process and products of these change steps, and to co-ordinate the 
activities of many agencies over a long duration to accomplish very difficult, complex and 
little understood tasks. This assessment concurs with a sound judgment on the scope of the 
optimal procedural design identified in the input evaluation. It calls for a collaborative 
arrangement between the participating agencies, in which the EPIE Institute and to a lesser 
extent the CEIMA, should assume primary responsibility for the subactivities of research, 
invention and training within the prospective program, at least during the period of execution 
of the change strategy. The two Australian agencies, together with EPIE Institute, would 
co-operate in the subactivities of design, construction, assembly, dissemination, 
demonstration, trial, installation, and institutionalisation. Because it has been identified that 
the neomobilistic setting, in which the prospective program is to be planned, structured, 
implemented and recycled involves large change being attempted in the face of low 
understanding by Australian educators about how to accomplish the change, the role of foreign 
organisations is likely to be critical for the program's success. 

The theoretical model presented in this chapter, however, cannot be substituted for the firsthand 
experiences of participation. Komoski, reflecting upon fifteen years of personal experience 
in planning, structuring, implementing and recycling the research, development, diffusion and 
adoption activities within the EPIE Institute, recalled the complexity of conducting these tasks 
within a neomobilistic setting in the following statement. 

"Evidence to this fact is that in 1967, with the help of Bob Stake and Tenry Denny, and contribu- 
tions by others like Jack Easely, Irving Morrissett, Ira Gordon, Fred Goodman, Maury Hash, Har- 
riet Taimage, and Mike Scriven, I started on what, at times, seems to be something akin to an eternal 
quest and at other times is more like a masochistic exercise. This exercise is known to the school 
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consumers it is designed to serve as the Educational Products Information Exchange Insti tuie; more 
commonly known as EPBE ("EPPY"). 

Whatever else it is, EPBE has been, and continues to be, an exercise in paradigm building that is 
continuously being shaped by the real world of instructional materials evaluation (the core of which 
is, for better or worse, textbook evaluation). And if this evolving paradigm has validity, it may 
come from the fact that its tangible output in the form of independendy researched information on 
textbooks and other instructional artifacts is valued enough by the people who evaluate and pur- 
chase textbooks to sustain its existence from year to year. This exercise in real world product 
evaluation has been - and continues to be - a bit like the proverbial *building the bicycle while 
riding it'. 

More accurately, it is very much an adaptive, evolutionary response to certain technological and 
ecological facts of educational and economic life in the twentieth century," 

(Komoski, P.K., 1982, Affecting the System through Productive Evaluation, Paper presented at 
the annual meeting of the American Educational Research Association, March 19-23, New York, 
1) 
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CHAPTER 6 



THE SELECTION OF A MODEL FOR 
INSTRUCTIONAL DESIGN ANALYSIS 



A second component of input to be considered now is the adoption of a model of instructional 
design analysis that may be appropriate to the Australian context. The intent in this chapter 
is to make this selection through examination of the range of such models and evaluative 
instruments available. The researcher endeavours to investigate this issue through 
consideration of two problems: firstly, whether the techniques of instructional design analysis 
and their sequence in practice possess generic characteristics universal to the eductional 
setting; and secondly, whether evidence can be determined on the adaptability of the 
techniques of instructional design analysis in a variety of geographical contexts. 



6.1 The Techiques and their Universality 

Komoski (1977) has developed" a schema of the materials' marketplace in order to illustrate 
the various stages of evaluation through which a material proceeds between its development 
and its use by students. Komoski (1985) recalled the development of this schema, as follows. 
"Unfortunately, tliose ultimate consumers - teachers anci students - have no easy means of com- 
municating to publishers and even their own districts that those materials are not as good as they 
need to be. As a textbook salesman said to me some years ago, *the reason for it is simple: the kids 
don't buy the books.' 

That pithy analysis has always struck me as such an accurate assessment of the overriding reality 
of the instructional materials* marketplace that after hearing it I decided to develop a schema of 
the marketplace (Komoski, 1977). I began by asking: if the markeQ)lace doesn't place its highest 
value on making sure that the products being developed and purchased are meeting the needs of 
the consumer, what competing factors are given a higher value? The result of my analysis was the 
Schema of the Materials' Marketplace (Figure 1). Clearly any such schema will miss some of the 
subtleties of the real worid. But many persons involved in that markeQ)lace have indicated that 
this schema is both valid and useful." 

(Komoski, P.K., 1985, ^Instructional Materials will not Improve until We change the System', 
Educational Leadership, 42: 7, 33). 
This schema consists of five characteristics: marketplace setting; predominant values; 
*evaluators'; evaluative criteria; and evaluative feedback. Five stages, also identified, allow 
for construction of a matrix to describe attributes.. The attributes described at each stage can 
then be used to determine the fundamental evaluative procedures occurring between a 
product's development and its use with learners. 
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The typology of the evaluative process identified by the researcher, through administering the 
measurement instrument to assess the application of techniques in the Tasmanian context, 
consists of five sequential stages: learner-based verification and revision, incorporating both 
formative (pilot-trial and pilot-test) and summative (field-trial and field-tests) evaluations, for 
product development; screening for product adoption; descriptive analysis and evaluation for 
product assessment; decision making for product selection; and decision-making for product 
implementation. The stages of the researcher's measurement instrument possess sindlarities 
to certain attributes of Komoski's schema. The similarities are identifiable, particularly in 
relation to the four stages of Komoski's characteristic, the *evaluators': developers/producers; 
screeners/adopters; selectors/prescribers; and users/learners. The similarities pertain to both 
the traits of each technique and the sequence in which the techniques are applied. 
It is predicted, therefore, that these techniques exhibit inherent traits irrespective of the 
particular contexts in the educational setting in which they are applied. In order to confirm 
this hypothesis, the traits of each of the five techniques are examined briefly as they have been 
identified in exemplary practices or described by researchers. 

The definition of the procedures, involved in applying techniques to provide learner-based 
verification and revision, has been exemplified best in the work of Brickell and Aslanian 
(1979). Building upon research conducted at the Southwest Educational Development 
Laboratory, Austin, Texas, they identified six techniques that can be applied to verify products 
at the stage of development: inspection; simulation; pilot-trial; pilot-test; field-trial; and 
field-test As described by Brickell and Aslanian, each technique exhibits a set of traits to 
verify the development of educational products. The traits of each technique appear to be 
governed by three variables: the degree of complexity of the trrits correlates with the stage 
of product development at which a technique is applied; the complexity of the traits correlates 
with the power of a technique; and the representativeness of sampling procedures correlates 
with the power of a technique. 

The universal application to the educational context of techniques for learner-based 
verification and revision is implied in the work of Brickell and Aslanian. The policies and the 
practices for learner-based verification and revision of instructional materials enacted by 
various state legislatures in the United States provides the evidence to verify the universality 
of these techniques. The issues of the controversy regarding learner-based verification, 
however, have not pertained to whether the techniques possess generic properties. Klein 
(1978) maintains that these issues centre upon the types of materials to which learner-based 
verification and revision should be applied, the appropriateness of methods of collecting and 
reporting data, and the determination of a relationship between learner-based verification and 
revision and quality in instructional materials. 

Similarly, criteria for screening instrucdonal materials for adoption have been defined by a 
number of national and state organisations in the United States, in particular, the American 
Library Association and the Association of American Publishers. Whilst there is extensive 
evidence of the ineffectual applications of criteria to screen instructional materials, there is 
little evidence to verify the generalisability of such criteria. 

The last three stages of the researcher's measurement instrument are based upon the functions 
of instructional design analysis - descriptive analysis, evaluation, and decision-making - 
described by Eraut et. al. (1975). Through comparative analysis of seven instruments, Eraut 
et. al. determined that each instrument includes a range, if not all, of these functions. This 
finding suggests that these functions, or techniques, possess universal applicability to the 
educational setting. 
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In conclusion, the consistency in descriptions reported on both the traits of the techniques and 
the sequence of their application suggests that these patterns may indeed be universal to the 
educational setting. This is represented by the apparent similarities between Komoski's 
schema of the materials' marketplace and the re ^^archer's more restrictive typology presented 
in the measurement instrument. Although it is not possible to provide empirical evidence to 
support this effect, these findings suggest that both the traits of each technique and the sequence 
of their application possess generic characteristics. 

6.2 The Models of Curriculum Development 

Two fundamental and contrasting models of curriculum development are indicated in the 
literature: the objectives model; and approaches that are opposed to the objectives model 
(Stenhouse, 1975; Lawton et al., 1978; Brady, 1983). To clarify discussion on the application 
of these models to instructional design analysis, each model is briefly discussed below. 
The objectives model has been of foremost importance to curriculum development since Tyler 
(1949) provided its first systematic account The objectives model presumes four broad 
principles: stating objectives; selecting learning experiences; organising learning 
experiences; and evaluation. The initial stage, stating objectives, from which the process of 
systematic curriculum development occurs, received detailed attention in a taxonomy of 
objectives for the cognitive domain by Bloom et al. (1956), and a taxonomy of objectives for 
the affective domain by Krathwolil et al. (1964). In contrast to this refinement of objectives, 
Taba (1962) provided the principal statement relating these principles to the practice of 
curriculum development through eight sequential steps: diagnosing needs; formulating 
objectives; selecting content; organising content; selecting learning experiences; organising 
learning experiences; evaluating; and checking for balance and sequence. 
Two main types of criticism have been levelled by curriculum theorists at the objectives model: 
criticism of the objectives model in toto; and criticism of specific aspects of the objectives 
model. This criticism has led curriculum theorists to postulate alternative approaches to 
curriculum development. Whereas Tyler emphasised the specification of objectives, such a 
sequential ordering of steps in curriculum planning is not recognised in these other approaches. 
Two main approaches might be said to support the latter principle: the process model; and 
tlie interactive model. 

Advocates of the process model stress that there should be no initial statement of objectives; 
greater emphasis should be placed upon methodology than content; both content and 
methodology are intrinsically valuable; and that evaluation should serve as a means to 
establish the worth of outcomes rather than to measure prespecified objectives. Peters (1956), 
first suggested the foundations for the process model by insisting that areas of knowledge in 
curriculum activities are intrinsic parts of the curriculum rather than means to ends, as they 
are treated in the objectives model. Stenhouse stressed that the process model is more 
appropriate to curricular areas which centre on knowledge and understanding whilst, at the 
same time, insisting that the objectives model is more appropriate to areas which emphasise 
information and skills. 

In the process model, it is presumed that a series of significant questions about a program must 
be discerned and answered, as it progresses. Stenhouse has provided principles upon which 
such questions should be established: four principles of planning which comprise selection 
of content, development of a teaching strategy, decisions on sequence, and diagnosis of student 
strengths and weaknesses, including applying the three preceding principles to individual 
cases; four principles of empirical study which comprise study of student progress, study of 
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teacher progress, establishing the feasibility of implementing the curriculum indifferent school 
contexts, learner contexts, environments and peer group situations, and providing information 
and explanation on the variability of effects in differing contexts and on different students; 
and one principle related to justification, the formulation of the intention or aim of the 
curriculum, which is open to scrutiny. 

Unlike the objectives model, a sequence between the different elements of the curriculum is 
not presumed in the interactive model. Brady (1983) contrasted the interactive model to the 
objectives model. Whereas the sequential ordering of the elements of the curriculum are 
presumed within the objectives model, the curriculum is viewed in the interactive model as a 
dynamic process between the elements, in which no element predominates. It is assumed, 
however, that changes to one curriculum element will affect the other elements. Cuniculum 
development can conraience with any one of the four curriculum elements, and cuniculum 
developers are not restricted in when and how they develop or modify the elements. 

6.2.1 The Adaptability of tiie Models to Different Contexts 

The intention now is to examine the instruments used to evaluate cuniculum materials and 
how they have been adapted to this purpose in different geographical contexts. Two 
instruments have had a predominant influence -upon the evaluation of cuniculum materials: 
firsdy, the instrument developed by Eash; and secondly, the Curriculum Materials Analysis 
System developed by the Social Science Education Consortium. Tyler's objectives model of 
cuniculum development is applied to instnictional design analysis in both instruments. 

6.2.1 .1 The Development of Eash's Instrument 

Eash (1972a) reported that the Office of Evaluation Research at the University of Illinois at 
Chicago Circle, working with the EPBE Institute, developed an instrument to evaluate 
curriculum materials during the late 1960's and early 1970's. First described by Eash (1970; 
1972b), this instrument consists of five parts: I Objectives; n Organisation of the Material 
(scope and sequence); m Methodology; IV Evaluation; and V Comment. Three types of 
scale are employed in this instrument: firsdy, a checklist is used to evaluate the characteristics 
of each construct; secondly, open-ended responses are used to analyse exceptions to these 
characteristics ric;scriptively; and thirdly, judgements of each construct are rendered on 
seven-point, graphic rating scales. 

Subsequently, Eash (1974) reported that this instrument was field-tested during its 
development with a group of twenty-five graduate students. The purpose of this pilot study 
was to determine inter-rater reliability. The subjects used this instrument to analyse two types 
of material under three conditions: firsdy, each subject analysed a sixth-grade reading 
package, individually; secondly, the subjects, grouped in seven teams, rated the same material 
collectively; and thirdly, the seven teams then rated a cuniculum bulletin on seventh-grade 
science collectively. The results indicated that, whilst the inter-rater reliability of each construct 
in the instrument exceeded .9, the inter-item reliability for each construct averaged .55. 
It has been mentioned earlier in this report that the instrument developed by Eash was adapted 
subsequendy by the EPIE Institute. Elliott (1985) has reported on the circumstances that led 
to this adaptation, and the nature of successive adaptations of the instrument by the EPIE 
Institute, as follows. 

"I can, however, throw some light on the transition between Hash's original instrument and the 
version of EPIE form A which you found outlined in EPBE Reports 73 and 74 - probably the most 
important transition in the development of EPIE form Aand all that followed firam it This crucial 
revision took place during a workshop in which Maury Eash was training school people in Califor- 
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nia to use his instrument (which EPBE had adopted) to analyse textbook programs in reading. As 
a direct result of feedback from participants in that workshop, the first version of EPEE form A was 
produced. 

The main problem involved in responding tO your request is that the evolution of the process 
EPIE use^J to analyse instructional materials is not documented except in successive revisions of 
our analysis instrument (once called EPIE form A) and indirectly in the introductions to the EPEE 
Reports. EPIE form A and its more recent descendants have been used primarily for gathering in- 
formation to be used by educationa' consumers rather than for research. While we have made 
revisions in our analysis format in response to feedback from educators who use our reports, we 
have not kept systematic records of ail the changes made and the reasons for them." 
Because Hash's instrument has been revised on successive cx:casions by the EPBE Institute to 
take account of the criticisms of educators, the EPIE instrument has evolved from Eash's 
instrument substantially in specific details. Eraut et al. have described Eash's instrument as 
being essentially evaluative, although combining evaluative and descriptive analytic functions 
in a limited way but failing to provide users witli decision-making information. The EPIE 
versions of this instrument have evolved to include a balance between each of the three 
functions - descriptive analysis, evaluation, and decision-making - so that users in a wide 
variety of American educational contexts are provided with information to select instructional 
materials that meet their particular local requirements and conditions. 
The substance of this evolutionary process can be conceived more readily, if the circumstances 
of the adoption of an EPEE version of Eash's instrument are described as they have occurred 
in a foreign context. Extension of the services of the EPBE Institute to the Canadian provinces 
of Alberta, Manitoba and British Columbia has led three departments of education in these 
provinces to form a collaborative organisation, the Canadian Exchange for Instrucdonal 
Materials Analysis (CEIMA). The ways that the CEIMA has adapted the method, the 
techniques and the practices of the EPIE Institute to suit Canadian conditions and requirements 
is discussed in the next section. 

6.2,1 -1.1 The Canadian Exchange for Instructional Materials Analysis (CEIMA) 

The development of a system to evaluate curriculum materials by tliree education departments 
of the western provinces of Canada has been widely reported (Wood, 1981; Wright 1983; 
and Syme, 1987). This development occurred, initially, as a result of membership with the 
EPBE Institute and, more recendy, through a co-operative venture between the three education 
departments, known as the Canadian Exchange for Instructional Materials Analysis (CEIMA). 
The need for a procedure to evaluate curriculum materials to support curriculum development 
was recognised by Canadian pro\dncial educational authorities in Manitoba, Alberta and 
British Columbia during the late 1970's. This need led to contact being made with the EPIE 
Institute, to which membership by subscription was then established, firsdy by Alberta 
Education in 1978, followed by Manitoba Education in 1979 and the Ministry of Education, 
British Columbia in 1980. As a result of this membership, the EPDB Institute trained Canadian 
education department personnel to become trainers of evaluators, provided a certification 
procedure and a bank of analyses, based upon the EPIE PRO/FTDLES. 
The procedure and process, adopted by the Canadian education departments to evaluate 
curriculum materials during the period of their membership with the EPIE Institute, were 
similar to those used by the EPBE Institute. Wright recounted that development of this 
procedure in British Columbia was initiated at first during 1978 through the establishment of 
a Management Committee by the Curriculum Development Branch. Then, materials selection 
committees were established during 1979, publishers were provided with guidelines during 
1980 for tendering proposals on the development of materials so that this innovative procedure 
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for adopting curriculum materials first fully came into practice during 1981. Wood reported 
that two evaluators, trained and certificated in the process, analysed each material using EPIE 
form A. A third person then synthesised the reports of the two analysts. Selection decisions, 
made by committees or departmental personnel, then followed. 

Syme has reported on the developments that led to the dissolution of the membership of the 
Canadian education departments with the EPIE Institute, as follows. 

"Over a period of approximately five years, we observed the EPEE instructional design instru- 
ment moving away from a descriptive format to a more evaluative one. The Canadian members 
also found that we were more interested in our analyses than those that were done in the United 
States. The three Provinces agreed to approach EPEE about withdrawing from the organisation 
and maintaining a more comprehensive and descriptive analysis of our own. EPIE agreed to this 
departure and the CEIMA (Canadian Exchange for Instructional Materials Analysis) began." 
This development was realised when British Columbia withdrew from membership of the 
EPIE Institute during 1982, followed by Alberta and Manitoba during 1983. The Canadian 
Exchange for Instructional Materials Analysis was formed by the three provinces at a joint 
meeting in January 1983. The formation of CEIMA had consequences that were more 
far-reaching than just the severance of membership and contact with the EPIE Institute. 
Commencing at this time, the education departments of the three provinces extended through 
CEIMA the organisational procedure for evaluating and selecting curriculum materials and 
adapted the instrument, which had been derived from the EPIE Institute. 
The adaptations to the instrument, originally derived from the EPIE Institute 's version of EPIE 
form A used during the 1970's, were made to meet two requirements: to restrict instructional 
design analysis mainly to a descriptive analytic function; and, in certain circumstances, to 
include extensive, but independently, determined descriptive analyses of readability and social 
considerations. The instrument currently used by the Canadian Exchange for Instructional 
Materials Analysis (1987) for instructional design analysis consists of six main parts: 1. 
Identification and Background; 2. Instructional Design Constructs (Intents - developer's 
rationale, learner goals, learner objectives, intents congruence - Contents - organisation, 
presentation, scope, extent of the content coverage, sequence, supporting materials, contents 
congruence - Methodology - presentation, characteristics of methodology, instructional model, 
teacher preparation, teacher training, methodology congruence - Means of Evaluation - 
presentation, purposes/functions, content/focus, means of evaluation congruence); 3. Other 
Considerations (Accuracy of Content, Currentness of Content, Balance/Bias of Content); 4. 
Instructional Design Congruence (Internal Congruence, External Congruence); 5. Readability 
Factors (Sentence Structure and Vocabulary, Idea Load) and 6. Production Factors (Product 
Development, Technical Considerations for A.V. Components, Packaging, Media 
Appropriateness, Analyst's Summary). Because this instrument is used by each of the three 
education departments, analyses can be shared by all members of CEIMA through a bank of 
analyses. 

Syme has suggested that the differences in the functions of descriptive analysis, evaluation 
and decision-making that have arisen between the approach used in Canada and that used by 
the EPIE Institute, have occurred because the evaluative function, as well as the 
decision-making function, has been largely encompassed in the roles of selection committees. 
Selections of materials made by selection committees are recommended to education 
departments for approval. 

6.2.1.1-1.1 Manitoba Education 

The details of the organisational procedure used in Manitoba have been published by Manitoba 
Education (1988). This document consists of three sections: procedures for the selection of 
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approved learning materials; continuing evaluation of new instructional materials; and 
guidelines for identifying bias in learning materials. Presented in the first section, the 
organisational procedure for the evaluation of curriculum materials in Manitoba consists of 
three stages: identification and screening; in-depth analysis (instructional design analysis); 
and selection for piloting or recommendation. 

Certain basic requirements are applied by the Curriculum Development and Implementation 
Branch, Manitoba Education to conduct this organisational procedure: publishers are 
informed by a learning materials consultant that their publications must comply with the 
procedures; materials selection sub-committees are appointed to conduct the three stages of 
the procedure; and all materials selection sub-committee members, consultants and other 
resource personnel must qualify as certified analysts of learning materials through a training 
program provided by the Cuniculum Development and Implementation Branch. 
Materials are screened for adoption by materials selection sub-committees, which are 
established for each subject area. Materials are screened by appraisal forms on criteria that 
relate to their sources, appropriateness to student groupmgs, potential for cuniculum match, 
balance in presentations of attitudes, and balanced reflections of the Canadian multicultural 
heritage. Instructional materials are screened into three categories: unsuitable submissions, 
which are rejected; submissions judged to be basic to the cuniculum, which are subjected to 
in-depth analyses; and submissions judged to be supplementary, which are subjected to an 
extensive reviews and curriculum matches. Two independent analysts apply the instrument 
used by the CEIMA for instructional design analysis to conduct in-depth analysis of each basic 
cuniculum material. Then a synthesis of both analyses is produced and the reading level 
analysed At the third stage, the materials selection sub-committees match the synthesis of 
each basic curriculum material to the cuniculum guide on seven criteria: whether it is 
student-oriented; whether its focus is Canadian; whether it presents different approaches or 
levels of sophistication; whether it serves a variety of functions on an inter-disciplinary basis; 
whether it meets specific needs; whether it reflects the variety of Manitoba*s ethnic groups; 
and whether it reflects inquiry-oriented methods. Selection leads to the basic cuniculum 
material being recommended for either final approval or pilot study. A supplementary material 
is selected on whether it is both cost-effective and not expendable. Once a material is approved, 
information on the in-depth analysis is provided to schools to assist their selection processes. 
Approved materials are retained on an approval list for three years. A procedure is applied to 
add or to remove materials on the approval list. Additions are made by presenting to publishers 
requests for submissions on a three-year cycle: language arts and social studies in the first 
year; mathematics and science in the second year, and other subjects in the third year. 
Analysts are required to apply guidelines for identifying bias in learning materials under two 
main categories: historical accuracy and balance; and comprehensiveness and unity. The 
former category is concerned with inclusiveness and concretenes? , and the latter category is 
concerned with laiiguage and realism of references to race, religion, sex, age, physical and 
mental capacities of portrayals in instructional materials. 

6.2.1 .1 .1 .2 Ministry of Education, Province of British Columbia 

The organisational procedure used in British Columbia consists of tiiree stages: instructional 
design analysis; curriculum match; and field-testing. Prior to the first stage, publishers 
submit their publications upon request to the Curriculum Development Branch. Instructional 
materials submitted by publishers are screened for adoption. Then, a coordinator of learning 
resources arranges for those prescribed and authorised materials adopted to be analysed for 
instructional design, readability, and social considerations by teachers who have qualified as 
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certified analysts in a training program conducted by the Curriculum Development Branch. 
Curriculum match is applied by curriculum committees to select instructional materials on the 
basis of certain guidelines: a preference is specified for materials developed at sources in 
British Columbia; minimum standards are specified for instructional design, readability and 
social considerations; and a preference is specified for non-consumable materials. Alternative 
guidelines are specified for matching materials developed in other Canadian provinces or at 
foreign sources. Those basic curriculum materials selected are recorded by the coordinator 
of learning resources and recommended for either field-tesring or are approved by the Ministry 
of Education. French materials are field-tested at the provincial level whilst materials in other 
subject areas are field-tested at a local level. Once basic curriculum materials are approved, 
an official list is sent to schools annually as the Prescribed and Authorized Learning Resources 
Catalogue, whilst analytic information is published in a Product Information booklet. 
Supplementary materials are listed in curriculum guides, included in resource books, and in 
the media resource guides and catalogue of the Provincial Educational Media Centre (PEMC). 
The details of this organisational procedure have been published by the Ministry of Education, 
Province of British Columbia (1987). This document consists of three parts: learning 
resources - definitions, status, distribution, submission procedures, selection procedures for 
prescribed and authorised learning resources, print learning resources developed under 
contract, selection procedures for print learning resources developed by other ministries and 
independent agencies, notification procedures for recommended supplementary; curriculum 
committees - curriculum committee membership, definition of responsibilities, and terms of 
reference; and an appendix. 

To assist different groups, such as evaluators and selection committee members, the Ministry 
of Education, Province of British Columbia (1982), has published a booklet to guide users in 
interpreting detailed analyses. This booklet consists of five chapters - an introduction, 
curriculum design, readability, social considerations, and a conclusion - and four appendices 

- glossary of terms (CEIMA), glossary of terms (readability), Ministry guidelines for selection, 
and subject areas. 

The Ministry of Education, Province of British Columbia (1983) has also published a booklet 
to assist publishers. This booklet consists of four chapters - an overview of the materials 
selection process, instructional design, readability, social considerations - and four appendices 

- Berber's levels of comprehension, glossary of terms (instructional design), glossary of terras 
(readability), and Ministry guidelines for selection. 

In addition to instructional desig.. ?jialysis, the Ministry of Education, Province of British 
Columbia has developed instruments to analyse readability and social considerations. These 
evaluations are done separately fix)m the instructional design analysis. 
Readability analysis is undertaken to match the reading level of a learner to instructional 
materials of an appropriate readability level. An evaluator uses the instrument to apply two 
measures of readability: firstly, surface measures, which identify the pl^ysical features of the 
written material that contribute to its ease or difficulty for reading, together with its appearance 
of ease or difficulty; and secondly, deep measures, which identify syntactic and semantic 
features of the written content- The instmment developed for readability analysis by the 
Ministry of Education, Province of British Columbia (1981a) comprises tliree main parts: 1. 
Identification and Background; 2. Surface Measures (Physical Features, Readability Formula 
Computations and Summary Statements); and 3. Deep Measures (Content Considerations - 
idea load, text style and structure, vocabulary load, and sentence structure and vocabulary - 
and Reading Purposes and Application - reading purposes). On the basis of this analysis, the 
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summary requires the evaluator to judge accordingly, the range of the grade level pJacement, 
student entry competencies and instructional setting. 

Social considerations analysis is undenaken to provide selection committees witli descriptive 
analyses of balanced portrayals, as well as biases, in curriculum materials. These include 
considerations of language usage, role portrayals of the sexes, belief systems, violence, sexual 
references, age portrayals, and any special considerations. In completing a social 
considerations analysis, an evaluator is invited to consider the author's tone, errors of omission, 
extent of references, frequency of occurrence, and setting of both the text and visu.^^! inclusions 
of the material in completing a descriptive analysis. The instrument developeu for social 
considerations analysis by the Ministry of Education, Province of British Columbia (1981b) 
comprises three parts: 1. Identification and Background; 2. Components of Social 
Considerations (Language Usage, Ethnic References, Role Portrayals of the Sexes, Reference 
to Belief Systems - religion, cults, philosophies, political views - Reference to Violence, Sexual 
References, Age Portrayals, References to Social Class, and Other Considerations); and 3. 
Analyst's Summary. The extensive nature of the social considerations analysis has been 
indicated by Syme, who stated iliat the text must be read in its entirity by tiie evaluator. 
Finally, Syme has indicated that attention is being given at present to the provision of analyses 
of audio-visual materials and computer courseware. 

"Analysis of non-print materials in British Columbia is carried out by the Provincial Education- 
al Media Centre. In the past, audio-visual materials and computer software have not been seen as 
integral to curriculum, but as supplementary elements. Currently, lhl<; focus is changing and we 
are working with PEMC to develop an analysis fomat for software and video which parallels the 
CEIMA format used for print resources and builds in the Social Considerations and Readability 
considerations of our other analyses". 

6.2.1 .2 The Curriculum Materials Analysis System 

Whereas Hash's instrument has been adapted tc \arying requirements and conditions in the 
educational context, the Curriculum Materials Analysis System, developed by the Social 
Science Education Consortium, has not been adapted to meet the needs of oth^r educational 
contexts. Rather, the Curriculum Materials Analysis Systern has provided the catalysis in 
another context to develop an evaluative instrument that applies an alternative approach to 
curriculum development. This development, wlilula occurred di^^ing the conduct of a 
significant project on the evaluation of curriculum materials, is now related. 

6.2.1 .2.1 Education Area, University of Sussex 

Between 1973 and 1975, the Centre for Educational Technology, Sussex University, Palmer, 
Brighton, Sussex, England, conducted a project, funded by the Volkswagen Foundation, to 
examine the evaluation of curriculum materials. The activities of the project, reponed b> 
Eraut et al. (1975), consisted of reading and consultation at an international level, producing 
analyses of different materials, developing a general analysis scheme, conducting an 
instructional program in instructional design analysis and one-week workshops on the 
evaluation of curriculum materials. 

Eraut et al. offered two reasons for adopting an international focus: first, to draw on the 
experiences of experts in curriculum development and instructional design analysis in other 
countries; and second, to support a belief that instructional design analysis is a significant 
means for communicating curriculum ideas at an international level. This would relate, they 
hoped, to the development of an approach which would be acceptable and usable 
internationally for evaluating both contextually-specific and culturally-specific instructional 
materials without imposing nationally-biased criteria. 




This focus was applied in two of the four decisions which influenced the direction of this 
project: firstly, a decision was made to restrict the project to the evaluation of instructional 
materials; secondly, a decision was made to adopt an evaluative instrument; thirdly, a 
decision was made to develop a generic instrument; and fourthly, a focus on decision-making 
was directed at a subject or content area level. In the context of this discussion, the application 
of an international focus is of greatest importance for the second and third decisions. 
Eraut et al. indicate that they had had considerable experience in using the Curriculum 
Materials Analysis System developed by the Social Science Education Consortium to conduct 
analyses of curriculum materials, and had extended the application of this instrument to 
subjects other than social studies. Dissatisfaction with the Curriculum Materials Analysis 
System was pardy responsible for the decision to develop their own instrument, titled the 
Sussex Scheme, which consists of five parLs: 1. Introduction; 2. Description and Analysis 
of Materials; 3. The Materials in Use; 4. Evaluation; and 5. Decision Making in a Specific 
Context, an optional part 

A characteristic feature of the Sussex Scheme is the developers' recommendation of a particular 
cuniculum model without incorporating it within the instrument, so that its use is not 
mandatory and can be substituted by other cuniculum models. In the interactive model 
recommended for curriculum development, the aims of a cuniculum program or material are 
expressed through four elements, none of which takes precedence, but operate through 
dynamic interaction: subject matter; objectives and outcomes; teaching, learning and 
communication methods; and assessment pattern. The developers argue that this allows a 
four-stage approach within part 2 and part 3 of the Sussex Scheme to be adopted This allows 
for explicit and realistic relationships to occur between author, analyst and user. 
Eraut et al. identified three essential functions of instruments used to evaluate cuniculum 
materials: descriptive analysis; evaluation; and decision-making. This instrument employs 
a separate evaluative ainction relating intents to differing standards and judgments so that the 
analyst is expected to express arguments both in support and in opposition to a cuniculum 
program or material. Unlike the Curriculum Materials Analysis System, the process for 
selecting curriculum materials is an integral function of the instrument developed by this 
project group. The developers of the Sussex Scheme adopt both selection and implementation 
decisions within the decision-making function so that users arc presented with pertinent 
evaluations of curriculum materials rather than compelling them to select materials to fit a 
particular curriculum design chosen beforehand. 

They indicate that one of the main purposes of their instrument, the Sussex Scheme, is to 
provide selection evidence which will allow those considering purchase or use of cuniculum 
materials io adopt, to adapt or to reject them. They argue that selectors should be presented 
with analyses of curriculum materials rather than being forced to decide upon the nature of 
the curriculum beforehand. Because selection is so closely linked to implementation, these 
writers believe that the last, optional section of their instrument. Decision Making in a Specific 
Context, should only be completed by a member of the user group. 
The current applications of the Sussex Scheme appear not to be documented. At the 
researcher's request, Sigurgeirsson interviewed Eraut (1988) on the current applications of the 
Sussex Scheme, both in Britain and in other countries. The substance of this interview is 
reported in an edited form, as follows. 

"As far aiJ I am aware, the Sussex Scheme has not been used by any official agency in Britain. 
However, it is being used in a number of places: in the third or fourth year of B.Ed. courses; in 
certificates for professional studies (in-service); and in the M.A. in curriculum studies at the Lon- 
don Institute of Education and elsewhere. It is also built into one of the Open University courses 
on curriculum evaluation. 



ERIC n2 



We continue to use it in M.A. courses at Sussex, sometimes with certain modifications. I run a 
one-week workshop for people working in schools, for people working in higher and further educa- 
tion, for a course we run for third-world educators, often from curriculum development centres. 

Outside Britain, I have recently conducted training workshops in Singapore and China. In Sin- 
gapore, I used it to train textbook reviewers and members of the Curriculum Development Institute 
of Singapore. In the People's Republic of China, I ran a training workshop last September for the 
People's Education Press. Since they publish about a billion textbooks a year and are currently 
in the process of revising them, the workshop A^as very timely. We had to work with interpreters, 
and also had a research student of ours from Hong Kong assisting. In the end we got some very 
useful results from it and I think they learned some useful things. We are in the process of get- 
ting the reports produced there translated from Chinese into English, as they provide a valuable 
resource of information about the nature of Chinese textbooks. 

I am beginning to think about approaching the workshops using the Sussex Scheme in a slight- 
ly different way. In particular, I intend talking about different methods of analysis, their strengths 
and wealcnesses, and inviting members of the workshop in the early stages to try out different 
methods on issues that they particularly want to investigate. Then, after they have gained some 
experience in trying different methods, to start working on the basis of the Scheme itself. By 'dif- 
ferent methods', I mean both various kinds of quantitative analysis and also various qualitative ap- 
proaches. One, for example, would be the quotation of an extract, followed by making a 
commentary on diat extract. Another would be treating a typology and then giving qualitative ex- 
amples to fit the typology." 

Eraut (1988) has developed ten such methods for collecting evidence on criteria in curriculum 
materials: quantitative survey (distribution of content, frequency and type of exercise, teaching 
points per lesson, photographs/diagrams per chapter); map of concepts, teaching points 
(content structure); inspection (organising features, explanation of purposes); excerpts with 
critiques (presentation, language style); detailed discussion of examples (logical 
development); reference to sentences in text (maccuracy, bias, design of exercises); other 
written sources (aims, alternative approaches, arguments on issues, theories); rating by experts 
(importance of topic, validity of interpretation); classroom trials or ratings by experienced 
teachers (motivation for pupils, effectiveness, suitability for homework); and teacher 
consultation (clarity of teachers' book). 

6.2.1.2.1.1 The Implementation of the Sussex Scheme in China 

Wu Yongxing (1988), Director of the Curriculum and Teaching Materials Research Lisdtuie, 
Beijing, People's Republic of China, has provided the following information on the workshop 
presented by Eraut at Beijing during September 1987, and on the prospective implementation 
of the Sussex Scheme in Chinese education. This statement is presented in an edited version 



"Dr Michael Eraut ran a training workshop last September in the People's Education Press (PEP) 
and the Curriculum and Teaching Materials Research Institute (CTMRI). The participants are 
very much interested in the Sussex approach of teaching materials analysis. They learned the 
theory of the approach first and then analysed some teaching materials, such as textbooks for 
Chinese, mathematics, geography, physics and chemistry. They completed a 150-page report 
Later, two articles that relate to the workshop were carried in our organisation's magazine. Cur- 
riculum, Teaching Material and Method, January of 1988. They fina that the Sussex Scheme is 
scientiHc and practical in teaching material analysis. 

In the next few years, we will be engaged ourselves in the development of teaching materials for 
the nine-year compulsory education level in the following order 1988 - development of experimen- 
tal teaching materials; 1989 - trial of the experimental teaching materials in local areas, then their 
revision and conduct of trial teaching again; 1990 - submission of the final version of the leaching 
materials for approval; and 1991 - application of the teaching materials on a national scale. 

We are going to use all means available to assist our research and development of the teaching 
materials, including the Sussex Scheme. But at the moment, we do not have any concrete ideas 
and means for the application of the Sussex Scheme." 
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6.2.1 .2.1 .2 The Adaptation and the Implementation of the Sussex Scheme in 
Iceland 

The Sussex Scheme is currently being implemented within the national education system of 
Iceland. This project is being conducted at present by Sigurgeirsson (1987) with support from 
the National Centre for Educational Materials at Reykjavik, Iceland. 
Iceland is both geographically isolated from Europe and peopled by an ethnically 
homogeneous, Icelandic-speaking community of 245,000. The education system is also 
characteristed by its uniformity. This system is almost entirely operated by the State through 
the Ministry of Education. The 230 schools, which are distributed across eight local education 
authorities, operate at two levels: a primary level of nine grades (ages 7 to 15 years), and a 
secondary level of four grades (ages of 16 to 20 years). 

Both the production and the dissemination of all curriculum materials, except those imported 
for the compulsory school level (K though lo grade 9), are handled by the National Centre for 
Educational Materials, a governmental authority founded in 1979. The introduction of 
educational videotapes, the operation of a Teachers Centre and the development of curriculum 
materials are the major undertakings of the National Centre for Educational Materials. 
The Sussex Scheme has been adapted for Icelandic use by Sigurgeirsson (1986a), who 
describes the process involved in the following terms. 

"The Sussex Scheme is then assessed with view to possible modifications: (1) by evaluating a 
cuiriculum analysis workshop where the Scheme was used, (2) by comparing it with recent litera- 
ture, and (3) by using the Scheme when analysing English and Icelandic curriculum materials". 

(Sigurgeirsson, L 1986a, Improving Curriculum Materials Development in Iceland through Cur- 
riculum Analysis, M, A. in Education thesis. University of Sussex, abstract) 
Although the major sections are retained within the Sussex Scheme, a considerable number 
of criteria have been added in the Icelandic version. 

The Icelandic version of the Sussex Scheme is being implemented through workshops 
conducted during 1987. It is anticipated that such workshops will provide the means to 
evaluate all curriculum materials cunentiy used in Icelandic schools, the means to develop a 
critical understanding of significant issues for development of curriculum materials, the means 
to encourage a greater degree of evaluation of curriculum materials m Iceland, the means to 
prevent uncritical adoptions of foreign curriculum materials, the means to develop skills in 
evaluating curriculum materials through preservice and inservice teacher education, the means 
to foster the professional development of developers of curriculum materials,.and the means 
to encourage educational organisations in Iceland to participate in such activities. 
Concuixent with the implementation of the Icelandic version of the Sussex Scheme, 
Sigurgeirsson (1986b) is also conducting research during 1987 and 1988 into the use of 
instructional materials in a stratified sample of Icelandic schools at the grades 4 through 6 
level. This research is designed to examine eight characteristics of instructional materials: 
their intrinsic characteristics; their patterns of use; their roles; their selection; perceived 
problems in their use; teachers' attitudes about their fundamental characteristics; their use by 
students; and students' attitudes about their fundamental characteristics. The research method 
is employing several techniques: observations of twenty classrooms in ten schools; the use 
of an interview schedule with teachers; and use of a quasi-experimental interview design with 
students. 
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6.3 Selection of a Model for the Australian Context 



The discussions of two.issues pertinent to the selection of a model for instructional design 
analysis have provided varying degrees of substantive evidence. The evidence suggests that 
the sequence in which these techniques are applied within an educational setting is governed 
by intrinsic factors of a universal type. Although this contention cannot be verijBed by 
empirical evidence, there is evidence that the methods, the techniques and the practices applied 
within this field of work can be adapted successfully in different geographical contexts. 
The intention now is to examine issues concerning the selection of an appropriate model of 
instructional design analysis for use in the Australian context. Such a selection is governed 
by two sets of factors: those that pertain to the curriculum; and those that pertain to the 
availability of resources. The former factor is now discussed in detail. 
Perhaps, Stenhouse (1975: 80-83) has provided the most convincing argument to guide the 
selection of the appropriate model of curriculum development. The gist of Stenhouse's 
argument centres on a pronouncement that the respective emphases placed upon either 
knowledge and understanding or information and skills should constitute the criterion for 
selecting a particular model. Stenhouse believed that when information and skills are more 
important, selection of the objectives model is appropriate and, on the other hand, when 
knowledge and understanding are more important, selection of the process model is 
appropriate. 

The prospective program entails essentially the collection, the synthesis and the diffusion of 
information. Subsidiary objectives, such as those concerned with inservice staff development 
and teacher education entail training, whilst others, such as implementing methods, techniques 
and practices or initiating research entail, principally, imparting knowledge and understanding. 
It is conjectured, therefore, that it is appropriate to select Tyler's objectives model for 
application to the methods, the techniques and the practices of this field of work, because such 
work is overwhelmingly concerned with developmg skills and providing information. 
Selection of an appropriate model of curriculum development for Australian education must 
also be informed by other considerations. The background to these considerations relates to 
antecedent conditions, for instance, the geographic, demographic and SGcioculmrai variables 
that influence Australian educ?non. Australian society remains predominantly a European 
enclave, in spite of increasing influences impinging from eastern Asia. Although isolated from 
other major concentrations of European populations in Europe and North America, both British 
and American approaches to curriculum development have influenced Australian educators. 
It is only recently that an indigenous school of curriculum thought has formed in Austialian 
educational circles, largely as a result of the foundation of the Curriculum Development 
Centre. Merging the prevailing British and American strands of curriculum thought into a 
pragmatic but eclectic approach, this school has sought through its foremost figure, Malcohn 
Skilbeck, to promote school-based curriculum development as an appropriate solution to many 
curriculum issues in contemporary Australian education. 

In spite of a clearer focus on curriculum theory by specialists, it is apparent today that a 
concensus does not prevail among the range of Australian educators on matters of curriculum 
development Furthermore, it is unclear whether evidence of their attitudes would be 
particularly helpful in inforaaing decision-makers on this issue. This situation is manifest in 
the practices of Australian educators, who are generally unaccustomed to analysing the 
curriculum in terms of its component elements. Consideration of these factors, however, does 
not seem to be useful for guiding selection of the appropriate model of curriculum 
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development, except to inform decision-makers that there appears to be no consensus on the 
matter, 

6.4 Conclusion 

Evidence that decision-makers can use to choose a model of instructional design analysis, that 
is appropriate for the Australian context, remains inconclusive. The discussion presented in 
this chapter has shown that empirical conclusions cannot be made that verify the universal 
applicability of the methods, the techniques and the practices employed within this field of 
work. 

On the other hand, the adaptation of these methods, techniques and practices in other 
^ geographical j:ontexts has been verified through the examples presented in this chapter. 
Furthermore, it can be determined that approaches to related educational problems applied in 
other educational settings can be assimilated into the methods, the techniques and the practices 
applied in this field of work. For instance, the examination has shown that the EPIE Institute 
has adopted approaches, such as the Degrees of Reading Power and the principles of 
curriculum alignment, originally applied by other educational organisations. This evidence 
confirms that the methods, the techniques and the practices applied in this field of work can 
be adjusted to requirements in other geographical contexts. 

The information on curriculum issues presented in this chapter would suggest informed 
judgement should support the selection of Tyler's objectives model of curriculum 
development Such a choice would be conducive to Australian education adapting those 
significant systems available for collecting, synthesising and diffusing information on 
instructional materials. 
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CHAPTER? 



THE CONCLUSION 



The purpose of this chapter is to present educational authorities with a set of recommendations 
for action in the problem area. The reconunendations refer to existing methods, techniques 
and practices to collect, to synihesise and to disseminate information on instructional materials 
in the case of the context evaluation and to the design of a system that applies reliable and 
valid techniques to collect, to synthesise and to disseminate information on instructional 
materials in the case of the input evaluation. The recommendations are specified on eight 
criteria stated by Stufflebeam et al. (1971a): institutional capability; program elements; 
management; personnel; funding; schedule; facilities; and communication, 

7.1 THE RECOMMENDATIONS 

The evaluation of the Australian context has provided evidence indicating that educational 
authorities should focus attention upon the following priorities in the problem area. 

The Context Evaluation 

. Institutional Capability 

Recommendation 1 

The capability of national educational agencies, such as the Curriculum Development Centre 
(CDC) and the Australian Schools Catalogue Information Service (ASCIS), to plan, to 
structure, to implement and to recycle a conimon program to collect, to synihesise and to 
disseminate information on instructional materials should be assessed. 

Recommendation 2 

The capability of State education departments to contribute to the planning, structuring, 
implementing and recycling of a national program to collect, to synthesise and to disseminate 
information on instructional materials should be assessed. 

Program Elements 

Recommendation 3 




The methods, techniques and practices that are applied at present to analyse instructional 
materials by Australian educational authorities should be reviewed with the intent that more 
reliable and valid methods, techniques and practices are adopted. 

Recornineadation 4 

The activities in existing programs to collect, to synihesise and to disseminate information on 
instructional materials, at both the national and the state levels, should be reviewed to identify 
duplicated services with the intent that services are rationalised. 

Recommendation 5 

The objectives and procedural designs of programs applied at present to collect, to analyse 
and to disseminate information on instructional materials by Australian educational authorities, 
should be reviewed. 

Mancigement 

Recommendation 6 

The qualifications and experiences of management staff working in the problem area should 
be assessed with the intent that such qualifications and experiences will meet, in future, criteria 
specified on the bases of requirements determined as essential in foreign settings. 

Personnel 

Recommendation 7 

The qualifications and experiences of professional personnel working in the problem area 
should be assessed with the intent that such qualifications and experiences will meet, in future, 
criteria specified on the bases of requirements determined as essential in foreign settings. 

Funding ■ 

Recommendation 8 

The extent of the current budget in personnel, equipment and maicrial, special services, travel, 
communications and space rental funded to collect, to synthesise and to disseminate 
information on instructional materials at both the national and state levels should be assessed. 

Schedule 

Recommendation 9 

The appropriateness, sufficiency and economy of the schedule for programs to collect, to 
synthesise and to disseminate information on instructional materials at both the national and 
state levels of Australian education should be reviewed with the intention of improving these 
attributes. 

Facilities 

Recommendation 10 

The appropriateness, sufficiency and economy of the facilities used in programs to collect, to 
synthesise and to disseminate information on instructional materials at both the national and 
state levels of Australian education should be reviewed with the intention of improving these 
attributes. 

Communication 
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. Recommendation 11 

The procedures for both publicising programs among practitioners and affording practitioners 
the opportunity to examine and to assess the operating qualities of existing programs should 
be reviewed. 

The Input Evaluation 

The evaluation of input for planned change in the Australian context has provided evidence 
indicating that educational authorities should focus attention upon the following priorities in 
the problem area. In this regard, educational authorities should consider the prospective 
program, described in Chapter 4, as the preferential plan. 

. Institutional Capability 

Recommendation 12 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should coUaborate to apply innovative methods, techniques and 
practices to collect, to synthesise and to disseminate information on instructional materials. 
Such collaboration is likely to strengthen the configuration between 
mission-program-component-products in internally shared activities. 

Recommendation 13 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should consult foreign organisations, such as the Educational 
Products Information Exchange (EPIE) Instimte and the Canadian Exchange for Instructional 
Materials Analysis (CEIMA), on the nature of input and co-operation that these organisations 
can contribute towards strengthening the configuration between 
mission-program-component-products in both internally and externally shared activities. 

. Program Elements 

Recommendation 14 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should adopt the specified objectives and scope of the principal 
aim of the prospective program stated in Chapter 4. 

Recommendation 15 

The Curriculum Development Centre (CDC) should explicate objectives to facilitate the 
integration of the National Software Co-ordination Unit (NSCU) and the prospective program. 

Recommendation 16 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should consult foreign organisations on planning, stmcmring, 
implementing and recycling the objectives of the prospective program, especially witii regard 
to research activities. 

. Management 

Recommendation 17 
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The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should give attention to the selection of a staff which has the 
requisite qualifications and experiences to manage the prospective program. 

Personnel 

Recommendation 18 

In collaboration with foreign organisations, the Curriculum Development Centre (CDC) and 
the Australian Schools Catalogue Information Service (ASCIS) should design and implement 
an in-service course to train personnel in the methods, techniques and practices to be applied 
in the collection, synthesis and disseminadon of information on instructional materials. 

Funding 

Recommendation 19 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should prepare a detailed budget for the prospective program, 
specifying the costs of personnel, equipment and materials, special services, travel, 
communications and space rental. 

Schedule 

Recommendation 20 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should prepare a schedule that specifies appropriate, sufficient 
and economical means to plan, to structure, to implement and to recycle the research, 
development, diffusion and adoption activities of the prospective program. 

Facilities 

Reeomrnenuation 21 

The Curriculum Development Centre (CDC) and the Australian Schools Catalogue 
Information Service (ASCIS) should assess the appropriateness, sufficiency and economy of 
their facilities for planning, structuring, implementing and recycling the research, 
development, diffusion and adoption activities of the prospective program. 

Communications 

Recommendation 22 

The Australian Schools Catalogue Information Service (ASCIS) should design a plan to create 
widespread awareness of the prospective program among practitioners and to afford 
practitioners the opportunity to examine and to assess the operating qualities of the prospective 
program. 
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APPENDIX A 



PAPER PRESENTED TO BOARD MEETING NO. 
10 OF THE AUSTRALIAN SCHOOLS 
CATALOGUE INFORMATION SERVICE 



SUBJECT: The evaluation of curriculum resources 

A. With the advent of ACIN and similar education databases and with the proposed review 
service, ASCIS has clearly moved significantly from its initial 'cataloguing only* phase. This 
is not only an outcome of the ASCIS Corporate Plan but is a development warmly welcomed 
by the Australian education community. It can be expected that, in the relatively near future, 
bibliographic records without 'value added* data v^dll be the exception rather than the rule. 

B. ASCIS is clearly no longer concerned merely with the efficient organisation of cuiriculum 
resources, although that remains a fundamental thrust of its services. It has moved into the 
field of carrying, indeed even creating, evaluative statements about cumcuium resources. It 
has thus begun a process by which it influen . :s the selection and the use of cuiriculum 
resources. 

C. The ASCIS enterprise of developing what is essentially a cuiriculum resources information 
database is driven by a conviction, not explicitly stated at least in these terms, that cuiriculum 
resources are a crucial component in the education proce.?s. By its quite proper inclusion of 
information about the content of curriculum resources in its database and its willingness for 
this information to be evaluative, ASCIS has declared that it has a role to play in the qualitative 
judgements which educators must make about the resources required for the development and 
realisation of cuiriculum in Australian schools. This role may be, indeed probably should be, 
chiefly that of a neutral communicator of such information. ASCIS has an interest, 
nevertheless, in attempting to ensure that such qualitative information is valuable to the schools 
it services. 

D. Australian approaches to the evaluation of curriculum resources have to date been relatively 
unsophisticated in comparison with other countries, notably the United States and Canada. In 
a survey of past and present practice in Australia by Michael Watt (unpublished thesis), certain 
critical assessments are made which conform to the observations of practitioners in the field. 
Evaluations are frequendy more descriptive than analytical. They tend to be made without 
explicit linkages to curriculum models, pedagogical theories, or even the likely relationship 
of the resources to objectives and outcomes in the classroom. Few evaluators have been trained 
for the task. The criteria for evaluation are not always explicit or discernible. Given ASCIS* 
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strategic interest in this matter as a carrier (or potential carrier) of evaluations, this criticism 
of Australian practice should be of concern to the ASCIS Board. 

E. This paper does not argue that ASCIS should itself play a direct role in remedying the faults 
in Australian evaluation practice identified above. Such action would go beyond its charter. 
It is argued, however, that ASCIS is entitled, perhaps obliged, in the interests of its users to 
encourage some other body to investigate the quality of the process of evaluating curriculum 
resources and to remind that body of ASCIS' interest in and capacity to deliver quality 
evaluations to Australian schools and educators. It is suggested that CDC may be the 
appropriate body to which the matter may be referred. 

F. The ASCIS Board is reminded that on two past occasions the writer has drawn attention to 
the American body, the Educational Products Information Exchange (EPIE) Institute. This 
Institute is extensively described in a research paper by Mr. Michael Watt of the Education 
Department of Tasmania. It has much experience and expertise in the development and use 
of criteria and instruments for the evaluation of curriculum resources. It also conducts training 
programs for educators in the principles and practice of curriculum resource evaluation. It 
also has a database of evaluations of all widely-used American curriculum resources, including 
computer software. In correspondence with Mr. Watt, the EPIE Institute has expressed an 
interest in working with Australian educators to address problems related to curriculum 
materials in Australian education. On this occasion and in light of this background paper, the 
Board may be willing to refer the matter to CDC. It would be for CDC to determine whether 
it was able and willing to open any negotiation with the EPIE Institute and to consult with Mr. 
Watt on his extensive background of research. The Board is merely asked to ^iignal an interest 
in the matter and to seek to interest another body in pursuing it 

G. Recommendation 

THAT ASCIS, because of its commitment and capacity to store and disseminate *value added' 
data to schools, and because of its legitimate interest in including appropriate evaluations of 
curriculum resources in such data, and because of its conviction that current Australian 
procedures for the evaluation of curriculum resources could and should be improved, and 
because it is aware of overseas sources of expertise in such matters, notably the EPIE Institute 
in the United States, resolve to request CDC to consider inviting the EPIE Institute to conduct 
a workshop for Australian edaucators on the evaluation of curriculum resources, with the 
objective of improving Australian practice. 

NOTE: If the Board were to adopt this resolution, a copy of this paper might well be included 
in the letter to CDC which conveys the request. The letter might also refer CDC to tiie work 
of Mr. Michael Watt on this subject. It would be up to CDC to decide whether or not to make 
contact with him. 



Glenn C. PuUen 
Tasmania 
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From Minutes of Board Meeting No. 10, p. 203 

8.26.3 Evaluation of curriculum resources 

Mr. PuUen tabled a paper suggesting the ASCIS Board request GDC to approach the 
Educational Products Information Exchange (EPIE) about conducting a workshop with a view 
to improving the Australian practice of evaluating curriculum resources. 

It was moved (Bahnisch/Macdonald): that ASCIS, because of its commitment and capacity 
to store and disseminate Value added' data to schools, and because of its legitimate interest 
in including appropriate evaluations of curriculum resources in such data, and because of its 
conviction that current Australian procedures could and should be improved, request CDC to 
conduct a workshop for Australian educators on the evaluation of curriculum resources, with 
the objective of improving Australian practice. 

Motion carried 10.217 
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A FORM FOR RATING THE FUNCTrONS OF 

TECHNIQUES 




Please follow the sequence of steps described below. 

1. Please read an interview record of these functions as they Dr3 described fur a particular site. 

2. Please read each question in the order presented on the rating form. 

3. Please write in your response for question 1 and question 2. 

4. For each other question in Section 1 through to Section 6, tick the box that best represents 
the function you believe is used at the site described in the written document. 

Section 1 : Background Information 

1. What is your name? 



2. What is the name of the agency assessed? 



3. To what type Ox'' material does this evaluation apply? 

(a) site-developed instructional material 

(b) site-developed educational equipment 

(c) commercially produced instructional material 

(d) commercially produced educational equipment 

Section 2: Learner-based Verification and Revision 

4. Does the process for product development include Jeamer-based verification and revision? 

(a) yes 

(b) no 

If you ticked 'no\ go on to Section 3. 

5. Pilot-trial. The term *pilot-irial* is used here to designate an initial, small-scale use of the 
product underconditions not so rigorous as those indicated by the term *pilot-test'. Apilot-trial 
is conducted before final development of the product has been completed. 




INSTRUCTIONS: 
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Are pilot-trials conducted? 

(a) yes 

(b) no 

6. Pilot-test. Similar tc a pilot-trial but more rigorous, including the use of at least one class 
group of students, the regular teacher using the pioduct, and the administration of foraial 
data-collection instruments to assess student learning. 

Are pilot-tests conducted? 

(a) yes 

(b) no 

7. Field-trial. Less rigorous than a field-test, a field-trial employs wide-scale use of inforaial 
methods of observation relying on teacher and student opinions about the utility and 
effectiveness of the product, stopping short of the requirements of good experimental design 
in selecting the sample. 

Are field-trials conducted? 

(a) yes 

(b) no 

8. Field-test. Similar to a field-trial but more rigorous, a field-test employs specification of 
the nature of the product to users, specification of the purposes of the field test to users, the 
wide-scale use of sampling procedures to select student groups, the administration of 
data-collection instruments to assess student learning, the training of those field-testing the 
product, and report writing of results for dissemination of information on the field-test to user 
groups. 

Are field-tests conducted? 

(a) yes 

(b) no " 

9. To what extent does that form of learner-based verification and revision used meet the 
description given? 

(a) yes, definitely 

(b) yes, probably 

(c) uncertain 

(d) no, probably not 

(e) no, definitely not 

Section 3: Screening for Adoption 

10. Does the process for adoption include screening? 

(a) yes 

(b) no 

If you ticked *no', go onto Section 4. 

11. Are informal procedures used? 

(a) yes 

(b) no 
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12. Is an appraisal form used? 

(a) yes 

(b) no 

Section 4; Descriptive analysis and evaluation 

1:>. Does the process for product assessment include descriptive analysis and evaluation? 

(a) yes 

(b) no 

If you ticked *nc\ go onto Section 5. 

14. Annotation. This term applies to a simple listing of characteristics of a material based upon 
descriptive criteria. 
Are annotations conducted? 

(a) '-es 

(b) no 



15. Descriptive analytical review. This term applies extended descriptive analysis of 
characteristics of a material. 

Are descriptive analytical reviews conducted? 

(a) yes 

(b) no 

16. Descriptive analytical and evaluative review. This term applies to extended descriptive 
analysis of characteristics of a material based upc:* descriptive and evaluative criteria. 

Are descriptive analytical and evaluative reviews conducted? 

(a) yes 

(b) no 

17. Instructional design analysis. This term applies to extended descriptive analysis of 
characteristics of a material based upon a model of curriculum development 

Are instructional design analyses conducted? 

(a) yes 

(b) no 

18. Instructional design analysis to establish internal congruence. This term applies to 
extended descriptive analysis of characteristics of a material based upon matching the elements 
within a model of curriculum development 

Are instructional design analyses conducted to establish internal congruence? 

(a) yes 

(b) no 

19. Instructional design analysis to establish external congruence. This term applies to 
extended descriptive analysis of characteristics of a material based upon a model of curriculum 
development that fits the material to an instructional program. 

Ai-e instructional design analyses conducted to establish external congruence? 




(a) yes 

(b) no 

20. To what extent does that form of descriptive analysis and evaluation used meet the 
description given? 

(a) yes, definitely 

(b) yes, probably 

(c) uncertain 

(d) no, probably not 

(e) no, definitely not 

Section 5: Decision-making for Selection 

21. Does the process for product selection include decision-making? 

(a) yes 

(b) no 

If you ticked *no\ go onto Section 6. 

22. Are informal procedures used? 

(a) yes 

(b) no 

23. Are rating scales used? 

(a) yes 

(b) no 

24. Are written briefs used? 

(a) yes 

(b) no 

Section 6: Decision-making for Implementation 

25. Does the process for product implementation include decision-making? 

(a) yes 

(b) no 

If you ticked *no\ stop here. 

26. Are informal procedures used? 

(a) yes 

(b) no 

27. Are written briefs used? 

(a) yes 

(b) no 

28. Are demonstrations used? 

(a) yes 

(b) no 
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APPENDIX C 



QUESTIONNAIRE AND INTERVIEW 

SCHEDULE 



Instruction: 

Please tick the appropriate box or write your response in the space provided. 

1. What is the name and address of your agency? 

Name of Agency 

Address 

Postcode Telephone [..J 

2. Do staff members of your agency trial with students curriculum materials that are being 
developed at your agency with the view to revising the materials? 

(a) yes 

(b) no 

3. Do staff members of your agency use an appraisal form, or some other informal process, to 
screen curriculum materials before the materials are used by teachers? 

(a) yes 

(b) no 

4. Do staff members of your agency develop and provide bibliographies, reviews or descriptive 
evaluations of curriculum materials for teachers? 

(a) yes 

(b) no 

5. Do staff members of your agency make reconraiendations on curriculum materials that affect 
their selection by teachers? 

(a) yes 

(b) no 
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6. Do staff members of your agency make recommendations to teachers on how particular 
curriculum materials can be implemented in their classrooms? 

(a) yes 

(b) no 

If you have answered 'yes' to any of questions 2, 3, 4, 5 and 6, please complete the remainder 
of this form, and then return it. If you have answered 'no' to each of questions 2, 3, 4, 5 and 

6. please stop here and return the form. 

7. Would you, or a designated staff member, be prepared to participate in an interview in order 
to describe the processes involved in such activities as occur at your agency? 

(a) yes 

(b) no 

8. Would you be prepared to supply written documentation describing the processes involved 
in such activities as occur at your agency? 

(a) yes 

(b) no 

(Such documentation could be enclosed when you return this questionnaire). 

9. What is the name of the contact person at your agency? 



Thank you for spending the time to complete this questionnaire. 
Please return the completed questionnaire to: 
Michael G. Watt, CCET, P.O. Box 256, NORTH HOBART, Tas. 7002. 



108 



APPENDIX D 



ERIC 



GUlDELmES FOR THE SPECIFICATION OF 
THE PROSPECTIVE PROGRAM 

Significance of focus 
. Priority 

Relate the mission or problem area to priorities in education. 

Guideline 1 

Describe the relationship to educational priorities recently expressed by the executive arm of 
government; the Department of Employment, Education and Training; or the National Board 
of Employment, Education and Training. 

In the absence of any demonstrable relationship to priorities expressed by government officials 
or other national figures, develop the argument that the selected problem area is nonetheless 
significant and merits support. 

Guideline 2 

Describe the relationship to educational priorities expressed by other leading individuals, 
organisations and agencies. 

Guideline 3 

Discuss the significance in social and economic terms, as well as from an educational point 
of view. Indicate whether the social and economic contributions are expected to be regional 
or national in eventual impact 

Guideline 4 

Identify the student population, demographic areas, and economic settings that are the targets 
of the proposed work. 

Guideline 5 

Discuss the likelihood that the work will continue to be significant in the future, even if 
priorities shift. 

. Amenability to a research and development approach 



Guideline 6 

Justify the assertion that tiie problems, needs, or opportunities identified can be dealt with 
satisfactorily through research and development activities and cannot be approached better by 
another means. 

Significance of outcomes 

Guideline 7 

Describe the significance of the prospective outcomes having direct benefits. 

Guideline 8 

Describe the significance of prospective outcomes in terms of timeliness and criticality. 

Uniqueness 

Guideline 9 

Explain how the creation of the proposed program will make a distinct or unique contribution 
to research and development capability in education - a capability not already available in 
existing institutions - or how it will significantly extend or improve what existing institutions 
can do. 

Guideline 10 

Describe the distinction in mission or problem area that will set the proposed program apart. 

Guideline 11 

Describe any distinct role tlie proposed program will play in its geographic area. 

Viability 

Planning capability 

Guideline 12 

Supply evidence that personnel who have guided and directed the development of the program 
plans can provide effective leadership in carrying them out 

. Political, legal, social, and moral viability 

Guideline 13 

Discuss the potential viability of the prospective program in political, legal, social and moral 
terms. 

. Cooperability-cooptability 

Guideline 14 

Describe any collaborative relationships with schools, universities, state departments of 
education, research and development organisations, or federal programs. Explain how these 
relationships will assist the institution in achieving its objectives. 

Guideline 15 

Explain what services other agencies will need to provide. Give evidence to certify their 
availability. 

Guideline 16 
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Describe the relationship between the proposed work and what has been accomplished 
previously or is under way elsewhere. 

. Parity 

Guideline 17 

Describe the procedure for establishing parity of representation from client groups. 

Guideline 18 

Describe the procedure for establishing parity of participation from key audiences. 

. Practicality 

Guideline 19 

Demonstrate that there is a reasonable balance between activities that accomplish 
organisationally defined objectives and exploratory projects that involve high risk but promise 
high payoff. 

Guideline 20. 

Describe a plan for publicising the program that will make the nature of its work clear, 
demonstrate its significance, and arouse interest in its potential. 

Guideline 21 

Specify the starting points, milestone points, and ending points for all programs and all 
program components. 

Personnel 

Guideline 22 

Explain how staff qualifications are relevant to carrying out the proposed program. 
Demonstrate how staff experiences are sufficiently varied and comprehensive. 

Guideline 23 

Describe the sources that will provide a continumg supply of regular personnel as well as 
consultants. 

Guideline 24 

Supply evidence that persons who engaged in planning and will continue as permanent staff 
members understand the plan, are in agreement with it, and are committ^^d to carrying it out. 

Adequacy of program elements 
. Objectives 

Guideline 25 

Specify defensible, clear, important objectives for the program. 

Supply evidence to justify selection of the objectives and to demonstrate that they were chosen 
reasonably from among a set of alternatives. 

State the objectives in operational terms, clearly specifying what the teraiinal product of each 
objective will be. For example, is the product to be new knowledge, some form of educational 
practice, or new instructional materials? Indicate the student populations, demographic areas, 
economic settings to be affected by achievement of the objectives. 
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Demonstrate that the objectives strike a favourable balance between addiessing critical needs 
and using available opportunities for meeting needs. 

State whether the achievement of the objectives is expected to make an incremental 
improvement, a sharp modification, or a complete change in the schools. 

Relevance of objectives to program and institutional goals 

Guideline 26 

Present the configuration of mission-program-component-products so that their completeness 
and their internal consistency will be apparent 

Demonstrate that the products of the program are necessary for the institution to achieve its 
mission. 

Demonstrate that programs are derived from the mission, that components constitute coherent 
programs, and that components will produce the anticipated products. 

. Adequacy of procedural design 

Guideline 27 

Specify defensible, clear, and relevant procedures for achieving the objectives. 

Supply evidence to justify the selection of-the procedux'es and to demonstrate that the choice 

among the available alternatives was reasonable. 

State the procedures clearly in operational terms. 

Show how the procedures are related to achieving the objectives. 

Describe the mechanisms for evaluating the program processes and the products, and 
demonstrate their adequacy for self-correction and redirection. 

Methodological adequacy 

Guideline 28 

Demonstrate that the methodology is appropriate, sufficient and economical for the work. 

. Appropriateness of schedule 

Guideline 29 

Demonstrate that the schedule is appropriate, sufficient and economical for the work. 

Adequacy of facilities 

Guideline 30 

Demonstrate that the facilities are appropriate, sufficient and econonrical for the work. 

Budget 

Guideline 31 

Supply a detailed budget. 

Specify the costs of personnel, equipment and materials, special services, travel, 
comirunications, and space rental. 

Demonstrate that the proposed budget is related to the proposed work. 
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APPENDIX E 



SURVEY OF ATTITUDES ON POTENTIAL 
FOREIGN INPUT INTO THE PROSPECTIVE 
PROGRAM IN AUSTEIALIAN EDUCATION 



Background 

This survey is intended to determine your opinions on the development of a relationship 
between your organisation and the prospective prograni in Australian education. 
The questionnaire employs a technique termed Policy Implications Analysis, reported by 
Doren L. Madey and A. Jackson Stenner, Tolicy Implications Analysis: A Method for 
Improving Policy Research and Evaluation', in C. B. Aslanian (ed.), 1981, Improving 
Educational Evaluation Methods: Impact on Policy (Sage Research Progress Series in 
Evaluation, vol. 11), Beverly Hills: Sage Publications, 23-39. 

Policy Implications Analysis is a technique intended to maximise the likelihood that an 
evaluation report will have an impact on decision-making. The Policy Implications Analysis 
technique is based upon two futures techniques: the Delphi Method; and Scenario Writing. 
The technique employs five steps: firstly, generation of hypothetical findings; secondly, 
preparation of a questionnaire to be administered to a selected panel of respondents; thirdly, 
administration of the questionnaire; fourthly, analysis of responses; and fifthly, use of the 
analysed responses to develop a sei of policy-relevant hypotheses. 
This survey instrument is comprised of two major sections: 

1 . the Introduction to the exercise, which familiarises you with the Policy Implications Analysis 
technique and the program under examination; and 

2. the Questionnaire. 

The questionnaire is divided into seven parts: 
Part I Background Information; 
Part n Institutional Capability (Preferential); 
Part ni Institutional Capability (Optional); 
Part IV Program Elements (Objectives 1 to 6); 
Part V Management - Personnel; 
Part VI Funding - Schedule; and 
Part Vn Facilities - Communication. 

Each part of the questionnaire, except the first part, contains an identical set of four 
components: firstly, a Hypothetical Finding; secondly, a sub-set of Guidelines on which you 
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respond through a graphic rating scale; and lastly, two sub-sets of open-ended questions, 
Significant Policy Implications and a Finding Statement. 

Responding to the Questionnaire 

It would be preferred if the responses were typed. 

Completing the questionnaire could take one person several hours. Several people, however, 
may respond independently to the questionnaire, each answering a particular part where 
expertise may be called upon. Respondents may also consult with their colleagues. This 
process is likely to expedite completion of the questionnaire. 

SECTION 1 : THE INTRODUCTION 

Step 1: Please familiarise yourself with the program under scrutiny by reading the enclosed 
copies of Chapter 2 and Chapter 4 of the evaluation report. 

Step 2: Please familiarise yourself with the technique of Policy Implications Analysis by 
reading the examples below, which include both items and responses. 
(Note: At this point, examples provided by Madey, D.L. and Stenner, A.J. ^Policy Implications 
Analysis: A Method for Improving Policy Research and Evaluation', pp 32-36 in Improving 
Educational Evaluation Methods: Impact on Policy by Aslanian, C.B. (ed.), copyright (c) 1981 
by Sage Publications, were reprinted by permission of Sage Publications, Inc. In accordance 
with the publisher's copyright requirements, the examples are not reprinted in Appendix E.) 

SECTION 2: THE QUESTIONNAIRE 

PART I: BACKGROUND INFORMATION 

Please write your response in the space provided or check the appropriate box. 

1. What is the name of your organisation? 

2. What is the position in the organisation of the person authorising these responses? 

3. Is your orga-^isation prepared to provide ihe information requested by completing the 
questionnaire? 

A. yes 

B. no, because it is the organisation's policy not to make such ' ' ^rmation available 

C. no, because of another reason (please specify) 

If you answered *yes' to Question 3, pler.se go onto Question 4 and then complete the 
Questionnaire.. If you answered *no' to Question 3, please stop here, detach Part I of tlie 
Questionnaire and return it by air mail to Mr. Michael Watt, Staff Development Section, 
Education Department of Tasmania, P.O. Box 256, North Hobart, Tasmania 7002, Australia. 

4. Does your organisation give permission to the author of this evaluation report to use the 
responses to develop a set of policy relevant hypotheses? 

A. no 

B. yes, without conditions 
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C. yes, but with conditions (please specify) 



Note: It is accepted by the researcher that opinions expressed in the responses are provided 
for research purposes only and are in no way binding upon the organisation providing this 
information. 

PART li: INSTITUTIONAL CAPABILITY (PREFERENTIAL) 

The items below in Part II refer to the following hypothetical finding. 

A. Hypothetical Finding 

As a preferential plan, the program is infused within a collaborative project conducted by a 
consortium, comprising a foreign organisation together with the Curriculum Development 
Centre (CDC) and the Australian Schools Catalogue Information Service (ASCIS). 
Responsibility for research and development activities of the project is appropriated tc che 
Curriculum Development Centre, although dependent to a considerable extent upon assistance 
from the foreign organisation. The implementation and diffusion of the project is carried out 
by th'j ASCIS, but supported by both the CDC and the foreign organisation. The role of the 
foreign organisation is largely advisory but its participation extends to sharing research 
activities, conducting a pilot study at state level, sharing data and other elements of common 
interest on a cooperative basis. 

B. Guidelines 

Below is a sub-set of five questions, numbered 1 to 5, each followed by a graphic rating scale. 
Please read each question and then circle the number that best fits your opinion of the 
relationship between the question and the hypothetical finding. 

1. To what degree is this finding within the purview of your organisation? 

Definitely within Definitely without 

123456789 10 

2. How much knowledge do you have of the general area addressed by this finding? 

Much knowledge Littie knowledge 

123456789 10 

3. To what degree does this finding correspond with your expectations? 

Expected Not expected 

123456789 10 

4. To what degree does this finding have immediate policy implications for a potential 
relationship between the prospective program and your organisation? 

To a large extent To a limited extent 

123456789 10 

5. Given that this finding reflects reality, is it stated in a concise and clear fashion? 

Communicates well Communicates poorly 

123456789 10 

C. Significant Policy Implications 

For each question in this sub-set, numbered 1 to 2, write in your response in the space provided. 
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1. What policy action(s) on the part of your organisation to support the prospective program 
might be precipitated by this finding? 

2. What further information would you need before taking action based upon this finding? 
D. Finding Statement 

For the item in this sub-set, write in your response in the space provided. 

Write a 'finding statement' that would support a recommendation on your part to drastically 
revamp the hypothetical finding for the prospective program. 

PART III: INSTITUTIONAL CAPABILITY (OPTIONAL) 

The items below in Part HI refer to the following hypothetical finding. 

A. Hypothetical Finding 

As an optional plan, the Australian Schools Catalogue Information Service (ASCIS) 
collaborates with a foreign organisation to develop and to implement the project for Australian 
education. The foreign organisation is entirely responsible for the research and development 
activities of the project, which includes a pilot study at the state level. Responsibility for the 
implementation and the dissemination of the project is carried out by the ASCIS, but supported 
by the foreign organisation. Because of the extensive degree of support required, the foreign 
organisation establishes and maintains an office in Australia. Therefore, the foreign 
organisation has scope to contribute both processes and products of its programs that are 
appropriate to the Australian educational community. 

B. Guidelines 

Below is a sub-set of five questions, numbered 1 to 5, each followed by a graphic rating scale. 
Please read each question and then circle the number that best fits your opinion of the 
relationship between the question and the hypothetical finding. 

1. To what degree is this finding within the purview of your organisation? 

Definitely within Definitely without 

123456789 10 

2. How much knowledge do you have of the general area addressed by this finding? 

Much knowledge Little knowledge 

123456789 10 

3. To what degree does this finding correspond with your expectations? 

Expected Not expected 

123456789 10 

4. To what degree does this finding have immediate policy implications for a potential 
relationship between tlie prospective program and your organisation? 

To a large extent To a limited extent 

123456789 10 
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5. Given that this finding reflects reality, is it stated in a concise and clear fashion? 
Communicates well Communicates poorly 

123456789 10 

C. Significant Policy Implications 

For each question in this sub-set, numbered 1 to 2, write in your response in the space provided. 

L What policy action(s) on the pan of your organisation to support the prospective program 
might be precipitated by this finding? 

2. What further information would you need before taking action based upon this finding? 

D. Finding Statement 

For the item in this sub-set, write in your response in the space provided. 

Write a 'finding statement' that would support a recommendation on your part to drastically 
revamp the hypothetical finding for the prospective program. 

PART IV PROGRAM ELEMENTS (OBJECTIVES 1 TO 6) 

The items below in Part IV refer to the following hypothetical finding. 

A. Hypothetical Finding 

It is decided that product producer information, product analyst information and product user 
information are collected in the program following the conduct of a pilot study. A decision is 
made that the technique of instructional design analysis based upon Tyler's objectives model, 
is applied to synthesise this information. Because of this decision, the EPIE instrument is 
adopted for data collections. It is also accepted that the instrument must be adapted to 
Australian requirements and conditions through revisions based upon criticisms received from 
educators. An in-service course to train, to certificate and to monitor analysts in the adopted 
technique of instructional design analysis is then developed and implemented. The intent of 
this course is to ensure high inter-rater reliability. Simultaneously, software, compatible to 
both the needs of the prospective program and the ASCIS standards, is developed through 
contract for the storage and the retrieval of data. In the meantime, it is decided that the 
synthesised data will be diffused through printed products until such a time as the data can be 
provided on-line to users. Assistance must be sought from a foreign organisation in order that 
each objective is adopted successfully. 

B. Guidelines 

Below is a sub-set of five questions, numbered 1 to 5, each followed by a graphic rating scale. 
Please read each question and then circle the number that best fits your opinion of the 
relationship between the question tnd the hypothetical finding. 

1. To what degree is this finding within the purview of your organisation? 




Definitely within Definitely without 

123456789 10 
X How much knowledge do you have of the general area addressed by this finding? 
Much knowledge Little knowledge 

123456789 10 

3. To what degree does this finding correspond with your expectations? 

Expected Not expected 

123456789 10 

4. To what degree does this finding have immediate policy implications for a potential 
relationship between the prospective program and your organisation? 

To a large extent To a limited extent 

123456789 10 

5. Given that this finding reflects reality, is it stated in a concise and clear fashion? 

Communicates well Communicates poorly 

123456789 10 

C. Significant Policy Implications 

For each question in this sub-set, numbered 1 to 2, write in your response in the space provided. 

1. What policy action(s) on the part of your organisation to support the prospective program 
might be precipitated by this finding? 

2. What further information would you need before taking action based upon this finding? 

D. Finding Statement 

For the item in this sub-set, write in your response in the space provided. 

Write a 'finding statement' that would support a recommendation on your part to drabdcally 
revamp the hypothetical finding for the prospective program. 

PART V: MANAGEMENT - PERSONNEL 

The items below in Pan V refer to the following hypothetical finding. 
A. Hypothetical Finding 

For the preferential plan, the conduct of the selection process for both management and 
professional staffs shows that all prospective Australian staff members are deficient in specific 
skills needed to run the program successfully. The management personnel appointed has 
gdned experience in directing the operation of a clearinghouse, but cannot draw upon prior 
experiences in managing these tasks when they relate to specific methods, techniques and 
practices in the program. As a means to overcome this constraint, the management decides to 
employ consultants fi"om a foreign organisation to advise on such matters. Whilst the 
professional staff possesses the requisite expertise in particu^^r content areas, it is also deficient 
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in knowledge and skills of these methods, techniques and practices. It is also decided that a 
foreign organisation should provide the trainers of the staff members of the progranL 

B. Guidelines 

Below is a sub-set of five questions, numbered 1 to 5, each followed by a graphic rating scale. 
Please read each question and then circle the number that best fits your opinion of the 
relationship between the question and the hypothetical findmg. 

1. To what degree is this finding within the purview of your organisation? 

Definitely within Definitely without 

123456789 10 

2. How much knowledge do you have of the general area addressed by this finding? 

Much knowledge Little knowledge 

123456789 10 

3. To what degree does this finding correspond with your expectations? 

Expected Not expected 

123456789 10 

4. To what degree does this finding have immediate policy implications for a potential 
relatioi.ship between the prospective program and your organisation? 

To a large extent To a limited extent 

123456789 10 

5. Given that this finding reflects reality, is it stated in a concise and clear fashion? 

Communicates well Communicates poorly 

123456789 10 

C. Significant Policy Implications 

For each question in this sub-set, numbered 1 to 2, write in your response in the space provided. 

1. What policy action(s) on the part of your organisation to support the prospective program 
niight be precipitated by this finding? 

2. What further information would you need before taking action based upon this finding? 

D. Finding Statement 

For the item in this sub-set, write in your response in the space provided. 

Write a Tmding statement' that would support a recommendation on your part to drastically 

revamp the hypothetical finding for the prospective program. 

PART VI: FUNDING - SCHEDULE 

The items below in Part VI refer to the following hypothetical finding. 

A. Hypothetical Finding 
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Because the program is to be operated by organisations at the national level of Australian 
education, the preferential plan is financed from federal ^^nds. These funds are sufficient to 
finance the costs of personnel, equipment and materials, special services, travel, 
communications and space rental. Such funds, however, are insufficient to support the research 
necessary to develop the program. The agencies responsible for maintaining the program, 
therefore, seek funds to support research from a foreign organisation. 
The provision of a schedule is dependent upon both the types of funding and the prior 
knowledge of the management in this field. Because the management staff cannot establish 
such a configuration, it lacks the capability to specify a schedule. Instead, the short-term 
objectives are implemented sequentially at points when judged appropriate. Hindsight proves 
that several of these judgements are incorrect. Because no provision has been made at points 
in the sequence to recycle activities, there is no way to correct unsatisfactory outcomes. 
Consequendy, the management decides to employ consultants from a foreign organisation to 
determine this configuration, and then provide a schedule. 



B. Guidelines 



Below is a sub-set of five questions, numbered 1 to 5, each followed by a graphic rating scale. 
Please read each question and then circle the number that best fits your opinion of the 
relationship between the question and the hypothetical finding. 

1. To what degree is this finding witliin the purview of your organisation? 

Definitely within Definitely without 

123456789 10 

2. How much knowledge do you have of the general area addressed by this finding? 

Much knowledge Littie knowledge 

123456789 10 

3. To what degree does this finding correspond with your expectations? 

Expected Not expected 

'^1 23456789 10 

4. To what degree does this finding have immediate policy implications for a potential 
relationship between the prospective program and your organisation? 

To a large extent To a limited extent 

123456789 10 

5. Given that this finding reflects reality, is it stated in a concise and clear fashion? 

Communicates well Communicates poorly 

123456789 10 



C. Significant Policy Implications 



For each question in this sub-set, numbered 1 to 2, write in yourresponse in he space provided. 

1. What policy action(s) on the part of your organisation to support the prospective program 
might be precipitated by this finding? 

2. What further information would you need before taking action based upon this finding? 



D. Findmg Statement 
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For the item in this sub-set, write in your response in the space provided. 
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Write a Tinding statement' that would support a recommendation on your part to drastically 
revamp the hypothetical finding for the prospective program. 

PART VII: FACILITIES - COMMUNICATION 

The items below in Part Vfl refer to the following hypothetical finding. 

A. Hypothetical Finding 

For the preferential plan, the project uses the facilities of both the Curriculum Development 
Centre and the Australian Schools Catalogue Information Service. Because it is found that 
the facilities of the Curriculum Development Centre are inadequate to support the research, it 
is decided to seek dependence upon the research facilities of a foreign organisation. Because 
the ASCIS has not developed, as yet, the on-line facilities to diffuse the program to users, 
assistance is sought from a foreign organisauon to specify the characteristics of the software. 
Communications are assumed by the ASCIS as part of its role in diffusing the program. The 
ASCIS plan developed to publicise the program during its developmental phase involves 
publication of a brochure with issues printed at regular intervals. The brochure is used to 
publicise timely developments at the point of adoption. During the operational phase, a 
newsletter is published at monthly or quarterly intervals to inform users of significant issues 
in the field and about the program. The newsletter includes self-report forms to gather data 
from users. As a means to maximise the effect of publicising the program, the mianagement 
staff decides to employ consultants from a foreign organisation to advise on such matters. 

B. Guidelines 

Below is a sub-set of five questions, numbered 1 to 5, each followed by a graphic rating scale. 
Please read each question and then circle the number that best fits your opinion of the 
relationship between the question and the hypothetical finding. 

1. To what degree is this finding within the purview of your organisation? 
Definitely within Definitely without 

123456789 10 
. 2. How much .knowledge do you have of the general area addressed by this finding? 
Much knowledge Little knowledge 

123456789 10 

3. To what degree does this finding correspond with your expectations? 

Expected Not expected 

1 23456789 10 

4. To what degree does this finding have immediate policy implications for a potential 
relationship between the prospective program and your organisation? 

To a large extent To a limited extent 

123456789 10 

5. Given that this finding reflects reality, is it stated in a concise and clear fashion? 

Communicates well Communica<.es poorly 
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C. Significant Policy Implications 

For each question in this sub-set, numbered 1 to 2, write in your response in the space provided, 

1. What policy action(s) on the part of your organisation to support the prospective program 
might be precipitated by this finding? 

2. What further information would you need before taking action based upon this finding? 

D. Finding Statement 

For the item in this sub-set, write in your response in the space provided. 

Write a 'finding statement' that would support a recommendation on your part to drastically 
revamp the hypothetical finding for the prospective program. 

Returning the completed Questionnaire 

Both the preliminary instructions and the Introduction can be detached from the Questionnaire. 

Please return the completed Questionnaire by air mail to: 

Mr Michael Watt, 

Staff Development Section, 

P.O. Box 256, 

NORTH HOB ART, 

TASMANIA, 7002, 

AUSTRALIA 
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GLOSSARY 



Section 1 : Learner-based Verification and Revision 

1.1 PiloMrial 

This term is used to designate an initial, small-scale use of the product under conditions not 
so rigorous as those indicated by the term 'pilot- test'. A pilot-trial is conducted before final 
development of the product has been completed. 

1.2 PiloWest 

Similar to a pilot-trial but more rigorous, including the use of at least one class group of 
students, the regular teacher using the product, and the administration of formal data-collection 
instruments to assess student learning. 

1 .3 Field-trial 

Less rigorous than a field-test, a field-trial employs wide-scale use of informal methods of 
observation relying on teacher and student opinions about the utility and effectiveness of the 
product, but stopping shon of the requirements of good experirnental design in selecting the 
sample. 

1.4 Field-test 

Similar to a field-trial, but more rigorous, a field-test employs specification of the nature of 
the product to users, specification of the purposes of the field-test to users, the wide-scale use 
of sampling procedures to select student groups, the administration of data-collection 
instruments to assess student learning, the training of those i1eld-test*ng the product, and report 
writing of results for dissemination of information on the field-test to user groups. 

Section 2: Screening for Adoption 

2.1 Informal Procedure 

This term encompasses a range of activities to screen instructional materials such as reading 
or informal consultations. 

2.2 Appraisal Form 

This is an ins trument embodying a set of agreed-upon criteria. It is administered by committee 
members as an initial step to screen materials. 

Section 3: Descriptive Analysis and Evaluation 

3.1 Annotat'on 
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This tenii applies to a simple listing of characteristics of a material based upon descriptive 
criteria. 

3.2 Descriptive Analytical Review 

This term applies to extended descriptive analysis of characteristics of a material. 

3.3 Descriptive Analytical and Evaluative Review 

This term applies to extended descriptive analysis of characteristics of a material based upon 
descriptive and evaluative criteria. 

3.4 Instructional Design Analysis 

This term applies to extended descript' ve analysis of characteristics of a material based upon 
a model of curriculum development. 

3.5 Instructional Design Analysis to Establish Internal Congruence 

This term applies to extended descriptive analysis of characteristics cf a material based upon 
matching the elements within a model of curriculum development. 

3.6 Instructional Design Analysis to Establish External Congruence 

This term applies to extended descriptive analysis of characteristics of a material based upon 
a model of curriculum development that fits the material to an instructional program. 

Section 4: Deci making for Selection 

4.1 Informal Procedure 

This term encompasses a range of activities to select instructional materials, such as scanning 
displays of materials, referring to publishers' catalogues, or consulting colleagues. 

4.2 Rating Scale 

This term applies to the use of a scale to rate recommendations of instructional materials for 
selection. 

4.3 Written BriRf 

This term rciers to a brief review aimed at judging the advantages and disadvantages of an 
instructional material. It is prepared by a consultant or prospective user to aid selection. 

Section 5: Decision-making for implementation 

5.1 Informal Procedure 

This term encompasses a range of activities to implement instructional materials, such as 
seminars for professional development. 

5.2 Written Brief 

This term refers to a brief review of an instructional material. It is prepared by a consultant 
or prospective user to aid implementation. 

5.3 Demonstration 

This term applies to the demonstration of an instructional material in the classroom to aid 
implementation. 
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